[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:31:15 dist-test-slave-q9h9 krb5kdc[6342](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:31:15 dist-test-slave-q9h9 krb5kdc[6342](info): set up 2 sockets
Aug 10 14:31:15 dist-test-slave-q9h9 krb5kdc[6342](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 14:31:16.501070 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.216s user 0.008s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:31:16 dist-test-slave-q9h9 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:31:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:31:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:31:16.664093 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36259
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:46709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:36259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:17.169034 6362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:17.169962 6362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:17.170217 6362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:17.179462 6362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:31:17.179674 6362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:17.179814 6362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:17.179998 6362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:31:17.180112 6362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:31:17.192258 6362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:46709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:36259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:17.194214 6362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:17.195978 6362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:17.218309 6372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:17.218917 6373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:17.219417 6375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:17.220036 6362 server_base.cc:1034] running on GCE node
I20260810 14:31:17.221294 6362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:17.223567 6362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:17.224787 6362 hybrid_clock.cc:648] HybridClock initialized: now 1786372277224694 us; error 89 us; skew 500 ppm
I20260810 14:31:17.225291 6362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:17 dist-test-slave-q9h9 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372277, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:17.233646 6362 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:31:17.236091 6362 webserver.cc:460] Webserver started at http://127.6.46.254:44863/ using document root <none> and password file <none>
I20260810 14:31:17.237128 6362 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:17.237300 6362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:17.237767 6362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:17.242499 6362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "e4c1d64c07eb448f91bae9f9b1a69fa9"
format_stamp: "Formatted at 2026-08-10 14:31:17 on dist-test-slave-q9h9"
server_key: "bbb39f66bdb3dabb003837f245261828"
server_key_iv: "f24db31195fd3e48b0ce5227a462b848"
server_key_version: "encryptionkey@0"
I20260810 14:31:17.243606 6362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "e4c1d64c07eb448f91bae9f9b1a69fa9"
format_stamp: "Formatted at 2026-08-10 14:31:17 on dist-test-slave-q9h9"
server_key: "bbb39f66bdb3dabb003837f245261828"
server_key_iv: "f24db31195fd3e48b0ce5227a462b848"
server_key_version: "encryptionkey@0"
I20260810 14:31:17.253314 6362 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260810 14:31:17.259716 6384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:17.261631 6362 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260810 14:31:17.261988 6362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "e4c1d64c07eb448f91bae9f9b1a69fa9"
format_stamp: "Formatted at 2026-08-10 14:31:17 on dist-test-slave-q9h9"
server_key: "bbb39f66bdb3dabb003837f245261828"
server_key_iv: "f24db31195fd3e48b0ce5227a462b848"
server_key_version: "encryptionkey@0"
I20260810 14:31:17.262352 6362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:17.285130 6362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:17.290737 6362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:17.291123 6362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:17.324446 6362 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:36259
I20260810 14:31:17.324548 6464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:36259 every 8 connection(s)
I20260810 14:31:17.326879 6362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:31:17.331887 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 6362
I20260810 14:31:17.332432 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:31:17.335194 6467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:17.335721 6331 external_mini_cluster.cc:1428] Setting key 9199b54c9799f0912a121dd86f0c3202
I20260810 14:31:17.357246 6467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9: Bootstrap starting.
I20260810 14:31:17.365046 6467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:17.367530 6467 log.cc:824] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:17.371944 6467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9: No bootstrap required, opened a new log
Aug 10 14:31:17 dist-test-slave-q9h9 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:17.386323 6467 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c1d64c07eb448f91bae9f9b1a69fa9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36259 } }
I20260810 14:31:17.386986 6467 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:17.387182 6467 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4c1d64c07eb448f91bae9f9b1a69fa9, State: Initialized, Role: FOLLOWER
I20260810 14:31:17.388314 6467 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c1d64c07eb448f91bae9f9b1a69fa9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36259 } }
I20260810 14:31:17.388863 6467 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:31:17.389209 6467 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:31:17.389396 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:17.341772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.6.46.254:36259)
0810 14:31:17.343290 (+ 1518us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:17.343306 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:17.344548 (+ 1242us) server_negotiation.cc:407] Connection header received
0810 14:31:17.359348 (+ 14800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:17.359367 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:17.359787 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:17.360105 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:17.365247 (+ 5142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:17.367460 (+ 2213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:17.369779 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:17.370572 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:17.370816 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:17.378850 (+ 8034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:17.378862 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:17.378880 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:17.378929 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:17.383610 (+ 4681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:17.384467 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:17.384475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:17.384483 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:17.384806 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:17.385301 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:17.385305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:17.385309 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:17.386303 (+ 994us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:17.386686 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:17.387394 (+ 708us) server_negotiation.cc:299] Negotiation successful
0810 14:31:17.387773 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1134,"thread_start_us":624,"threads_started":1}
I20260810 14:31:17.389432 6467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:17.393630 6467 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c1d64c07eb448f91bae9f9b1a69fa9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36259 } }
I20260810 14:31:17.394686 6467 leader_election.cc:302] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4c1d64c07eb448f91bae9f9b1a69fa9; no voters:
I20260810 14:31:17.395781 6467 leader_election.cc:288] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:31:17.396045 6476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:17.397418 6476 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [term 1 LEADER]: Becoming Leader. State: Replica: e4c1d64c07eb448f91bae9f9b1a69fa9, State: Running, Role: LEADER
I20260810 14:31:17.398460 6476 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c1d64c07eb448f91bae9f9b1a69fa9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36259 } }
I20260810 14:31:17.399385 6467 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:31:17.404342 6479 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4c1d64c07eb448f91bae9f9b1a69fa9. Latest consensus state: current_term: 1 leader_uuid: "e4c1d64c07eb448f91bae9f9b1a69fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c1d64c07eb448f91bae9f9b1a69fa9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36259 } } }
I20260810 14:31:17.404441 6478 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4c1d64c07eb448f91bae9f9b1a69fa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4c1d64c07eb448f91bae9f9b1a69fa9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36259 } } }
I20260810 14:31:17.404810 6479 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:17.405004 6478 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:17.406597 6484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:31:17.417263 6484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:31:17.434458 6484 catalog_manager.cc:1347] Generated new cluster ID: 4e7053e0c6854907b2391ddff14a4646
I20260810 14:31:17.434583 6484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:31:17.452194 6484 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:31:17.454085 6484 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:31:17.463215 6484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4c1d64c07eb448f91bae9f9b1a69fa9: Generated new TSK 0
I20260810 14:31:17.464061 6484 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:31:17.521075 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36259
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:46709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:17.955861 6502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:17.956210 6502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:17.956349 6502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:17.964880 6502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:17.964977 6502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:17.965101 6502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:31:17.975600 6502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:46709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:17.976681 6502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:17.977911 6502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:17.992430 6509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:17.992669 6508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:17.992473 6512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:17.993845 6502 server_base.cc:1034] running on GCE node
I20260810 14:31:17.994408 6502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:17.995292 6502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:17.996524 6502 hybrid_clock.cc:648] HybridClock initialized: now 1786372277996456 us; error 42 us; skew 500 ppm
I20260810 14:31:17.996814 6502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:17 dist-test-slave-q9h9 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372277, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:18.000550 6502 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:31:18.001955 6502 webserver.cc:460] Webserver started at http://127.6.46.193:38111/ using document root <none> and password file <none>
I20260810 14:31:18.002588 6502 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:18.002774 6502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:18.003140 6502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:18.005884 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "479a25df570841608c814d6d1d896cfc"
format_stamp: "Formatted at 2026-08-10 14:31:18 on dist-test-slave-q9h9"
server_key: "c6f3647787a533c454d15a95f5a2bbcc"
server_key_iv: "7bb5aa702836458279d9721e1f301ce4"
server_key_version: "encryptionkey@0"
I20260810 14:31:18.006840 6502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "479a25df570841608c814d6d1d896cfc"
format_stamp: "Formatted at 2026-08-10 14:31:18 on dist-test-slave-q9h9"
server_key: "c6f3647787a533c454d15a95f5a2bbcc"
server_key_iv: "7bb5aa702836458279d9721e1f301ce4"
server_key_version: "encryptionkey@0"
I20260810 14:31:18.012449 6502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 14:31:18.015769 6521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:18.017163 6502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:31:18.017340 6502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "479a25df570841608c814d6d1d896cfc"
format_stamp: "Formatted at 2026-08-10 14:31:18 on dist-test-slave-q9h9"
server_key: "c6f3647787a533c454d15a95f5a2bbcc"
server_key_iv: "7bb5aa702836458279d9721e1f301ce4"
server_key_version: "encryptionkey@0"
I20260810 14:31:18.017539 6502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:18.046803 6502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:18.050660 6502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:18.050947 6502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:18.051937 6502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:18.053507 6502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:18.053596 6502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:18.053761 6502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:18.053833 6502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:18.102437 6502 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:44779
I20260810 14:31:18.102443 6707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:44779 every 8 connection(s)
I20260810 14:31:18.104044 6502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:31:18.110350 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 6502
I20260810 14:31:18.110636 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:31:18.111325 6331 external_mini_cluster.cc:1428] Setting key ecd94e5dad8f19ee7efb70bfdf8891e6
Aug 10 14:31:18 dist-test-slave-q9h9 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372277, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:18.126596 6711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:18.107329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:45667 (local address 127.6.46.254:36259)
0810 14:31:18.107939 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:18.107944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:18.109127 (+ 1183us) server_negotiation.cc:407] Connection header received
0810 14:31:18.111460 (+ 2333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:18.111466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:18.111545 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:18.111636 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:18.113488 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:18.114511 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:18.116118 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:18.116492 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:18.116597 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:18.120155 (+ 3558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:18.120164 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:18.120169 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:18.120209 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:18.123727 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:18.124438 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:18.124442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:18.124444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:18.124559 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:18.124949 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:18.124954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:18.124957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:18.125225 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:18.125355 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:18.125991 (+ 636us) server_negotiation.cc:299] Negotiation successful
0810 14:31:18.126237 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":326,"threads_started":1}
I20260810 14:31:18.127574 6713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:18.107968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36259 (local address 127.6.46.193:45667)
0810 14:31:18.108944 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:18.108993 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:18.111208 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:18.111890 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:18.111898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:18.112507 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:18.113325 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:18.113345 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:18.114651 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:18.114655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:18.115978 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:18.115990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:18.116701 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:18.116710 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:18.116909 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:18.117513 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:18.117535 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:18.119927 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:18.123892 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:18.123898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:18.123909 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:18.124329 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:18.124664 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:18.124669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:18.124672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:18.124845 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:18.125345 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:18.125353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:18.125653 (+ 300us) client_negotiation.cc:769] Sending connection context
0810 14:31:18.125879 (+ 226us) client_negotiation.cc:240] Negotiation successful
0810 14:31:18.126416 (+ 537us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":468,"threads_started":1}
I20260810 14:31:18.129245 6708 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36259
I20260810 14:31:18.129652 6708 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:18.130602 6708 heartbeater.cc:511] Master 127.6.46.254:36259 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
I20260810 14:31:18.134323 6408 ts_manager.cc:194] Registered new tserver with Master: 479a25df570841608c814d6d1d896cfc (127.6.46.193:44779)
I20260810 14:31:18.136510 6408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:45667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:31:18.154978 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36259
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:46709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:18.578388 6717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:18.578645 6717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:18.578776 6717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:18.587111 6717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:18.587221 6717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:18.587368 6717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:31:18.597611 6717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:46709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:18.598863 6717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:18.600292 6717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:18.615506 6723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:18.616351 6724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:18.616602 6728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:18.617251 6717 server_base.cc:1034] running on GCE node
I20260810 14:31:18.617815 6717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:18.618819 6717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:18.620040 6717 hybrid_clock.cc:648] HybridClock initialized: now 1786372278619983 us; error 45 us; skew 500 ppm
I20260810 14:31:18.620347 6717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:18 dist-test-slave-q9h9 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372278, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:18.623857 6717 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:31:18.625402 6717 webserver.cc:460] Webserver started at http://127.6.46.194:46727/ using document root <none> and password file <none>
I20260810 14:31:18.626188 6717 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:18.626302 6717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:18.626596 6717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:18.629575 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "71c9156ee36443cfba684fc7e7057e7e"
format_stamp: "Formatted at 2026-08-10 14:31:18 on dist-test-slave-q9h9"
server_key: "ce4c390fe4ef9242e8d4b9778b266d04"
server_key_iv: "21c8811d885bfc283d7a5af87eab791f"
server_key_version: "encryptionkey@0"
I20260810 14:31:18.630468 6717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "71c9156ee36443cfba684fc7e7057e7e"
format_stamp: "Formatted at 2026-08-10 14:31:18 on dist-test-slave-q9h9"
server_key: "ce4c390fe4ef9242e8d4b9778b266d04"
server_key_iv: "21c8811d885bfc283d7a5af87eab791f"
server_key_version: "encryptionkey@0"
I20260810 14:31:18.635902 6717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 14:31:18.639508 6742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:18.640735 6717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 14:31:18.640885 6717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "71c9156ee36443cfba684fc7e7057e7e"
format_stamp: "Formatted at 2026-08-10 14:31:18 on dist-test-slave-q9h9"
server_key: "ce4c390fe4ef9242e8d4b9778b266d04"
server_key_iv: "21c8811d885bfc283d7a5af87eab791f"
server_key_version: "encryptionkey@0"
I20260810 14:31:18.641085 6717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:18.657146 6717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:18.660863 6717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:18.661126 6717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:18.661974 6717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:18.663251 6717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:18.663327 6717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:18.663444 6717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:18.663508 6717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:18.714814 6717 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:33211
I20260810 14:31:18.714905 6920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:33211 every 8 connection(s)
I20260810 14:31:18.716240 6717 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:31:18.721457 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 6717
I20260810 14:31:18.721769 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:31:18.722656 6331 external_mini_cluster.cc:1428] Setting key e4661325cec5b868c2fe935da10c472e
Aug 10 14:31:18 dist-test-slave-q9h9 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372278, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:18.742272 6926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:18.719034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:55619 (local address 127.6.46.254:36259)
0810 14:31:18.719501 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:18.719506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:18.721025 (+ 1519us) server_negotiation.cc:407] Connection header received
0810 14:31:18.724381 (+ 3356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:18.724386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:18.724465 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:18.724569 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:18.727288 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:18.728138 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:18.730408 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:18.730936 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:18.731037 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:18.734925 (+ 3888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:18.734932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:18.734937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:18.734975 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:18.738775 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:18.739592 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:18.739598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:18.739601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:18.739861 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:18.740409 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:18.740414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:18.740418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:18.740802 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:18.740971 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:18.741625 (+ 654us) server_negotiation.cc:299] Negotiation successful
0810 14:31:18.741866 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":223,"threads_started":1}
I20260810 14:31:18.743132 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:18.719707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36259 (local address 127.6.46.194:55619)
0810 14:31:18.720828 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:18.720892 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:18.724124 (+ 3232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:18.724853 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:18.724864 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:18.725910 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:18.727090 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:18.727119 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:18.728304 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:18.728309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:18.730265 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:18.730279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:18.731160 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:18.731167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:18.731429 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:18.732108 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:18.732135 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:18.734585 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:18.738906 (+ 4321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:18.738913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:18.738924 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:18.739452 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:18.740007 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:18.740012 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:18.740016 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:18.740256 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:18.740966 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:18.740972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:18.741270 (+ 298us) client_negotiation.cc:769] Sending connection context
0810 14:31:18.741483 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 14:31:18.741859 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":576,"threads_started":1}
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
I20260810 14:31:18.744570 6922 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36259
I20260810 14:31:18.744843 6922 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:18.745831 6922 heartbeater.cc:511] Master 127.6.46.254:36259 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
I20260810 14:31:18.747292 6408 ts_manager.cc:194] Registered new tserver with Master: 71c9156ee36443cfba684fc7e7057e7e (127.6.46.194:33211)
I20260810 14:31:18.748368 6408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:55619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:31:18.770861 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36259
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:46709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:31:19.140383 6708 heartbeater.cc:503] Master 127.6.46.254:36259 was elected leader, sending a full tablet report...
W20260810 14:31:19.227519 6930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:19.227844 6930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:19.228018 6930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:19.236197 6930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:19.236310 6930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:19.236507 6930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:31:19.247408 6930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:46709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:19.248589 6930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:19.249886 6930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:19.263860 6941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:19.263877 6947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:19.263933 6942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:19.265260 6930 server_base.cc:1034] running on GCE node
I20260810 14:31:19.265933 6930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:19.266877 6930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:19.268070 6930 hybrid_clock.cc:648] HybridClock initialized: now 1786372279268028 us; error 52 us; skew 500 ppm
I20260810 14:31:19.268368 6930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:19 dist-test-slave-q9h9 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372279, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:19.272383 6930 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:31:19.274008 6930 webserver.cc:460] Webserver started at http://127.6.46.195:45975/ using document root <none> and password file <none>
I20260810 14:31:19.274904 6930 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:19.275127 6930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:19.275470 6930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:19.278733 6930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "7d71f6e628d5455eb1956902b2f7892c"
format_stamp: "Formatted at 2026-08-10 14:31:19 on dist-test-slave-q9h9"
server_key: "749150f5d94fb167a94e56dca3aa4164"
server_key_iv: "351ea1f18ab9c96a3eebefccae28b096"
server_key_version: "encryptionkey@0"
I20260810 14:31:19.279695 6930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "7d71f6e628d5455eb1956902b2f7892c"
format_stamp: "Formatted at 2026-08-10 14:31:19 on dist-test-slave-q9h9"
server_key: "749150f5d94fb167a94e56dca3aa4164"
server_key_iv: "351ea1f18ab9c96a3eebefccae28b096"
server_key_version: "encryptionkey@0"
I20260810 14:31:19.285665 6930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 14:31:19.289219 6961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:19.290462 6930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 14:31:19.290616 6930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "7d71f6e628d5455eb1956902b2f7892c"
format_stamp: "Formatted at 2026-08-10 14:31:19 on dist-test-slave-q9h9"
server_key: "749150f5d94fb167a94e56dca3aa4164"
server_key_iv: "351ea1f18ab9c96a3eebefccae28b096"
server_key_version: "encryptionkey@0"
I20260810 14:31:19.290791 6930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:19.305776 6930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:19.310642 6930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:19.310988 6930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:19.312206 6930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:19.313669 6930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:19.313786 6930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:19.313915 6930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:19.314007 6930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:19.369196 6930 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:32933
I20260810 14:31:19.369333 7152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:32933 every 8 connection(s)
I20260810 14:31:19.370833 6930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:31:19.379433 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 6930
I20260810 14:31:19.379694 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:31:19.380395 6331 external_mini_cluster.cc:1428] Setting key 5ebb7adff3659b4d83647cf689806b4e
Aug 10 14:31:19 dist-test-slave-q9h9 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372279, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:19.395270 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:19.373712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:47163 (local address 127.6.46.254:36259)
0810 14:31:19.374499 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:19.374503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:19.375655 (+ 1152us) server_negotiation.cc:407] Connection header received
0810 14:31:19.378365 (+ 2710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:19.378371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:19.378451 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:19.378546 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:19.381116 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.382366 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.384524 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.385286 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.385411 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:19.389317 (+ 3906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:19.389322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:19.389326 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:19.389365 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:19.392330 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:19.393062 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:19.393066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:19.393068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:19.393184 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:19.393701 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:19.393704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:19.393706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:19.393954 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:19.394125 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:19.394794 (+ 669us) server_negotiation.cc:299] Negotiation successful
0810 14:31:19.394984 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":461,"threads_started":1}
I20260810 14:31:19.396538 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:19.374263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36259 (local address 127.6.46.195:47163)
0810 14:31:19.375479 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:19.375543 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:19.378081 (+ 2538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:19.378876 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:19.378907 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:19.379848 (+ 941us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:19.380925 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:19.380955 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.382530 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.382536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:19.384390 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:19.384404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.385607 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.385617 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:19.385838 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:19.386542 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:19.386564 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:19.389108 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:19.392458 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:19.392465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:19.392477 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:19.392945 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:19.393309 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:19.393313 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:19.393316 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:19.393610 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:19.394081 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:19.394087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:19.394471 (+ 384us) client_negotiation.cc:769] Sending connection context
0810 14:31:19.394761 (+ 290us) client_negotiation.cc:240] Negotiation successful
0810 14:31:19.395180 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"thread_start_us":655,"threads_started":1}
I20260810 14:31:19.398424 7153 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36259
I20260810 14:31:19.398736 7153 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:19.399554 7153 heartbeater.cc:511] Master 127.6.46.254:36259 requested a full tablet report, sending...
I20260810 14:31:19.401177 6408 ts_manager.cc:194] Registered new tserver with Master: 7d71f6e628d5455eb1956902b2f7892c (127.6.46.195:32933)
I20260810 14:31:19.402199 6408 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:47163
I20260810 14:31:19.409600 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:31:19 dist-test-slave-q9h9 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372279, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:31:19 dist-test-slave-q9h9 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372279, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:31:19.460263 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:19.441373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.6.46.193:44779)
0810 14:31:19.442410 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:19.442419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:19.442523 (+ 104us) server_negotiation.cc:407] Connection header received
0810 14:31:19.442710 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:19.442717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:19.442974 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:19.443207 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:19.445163 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.446520 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.448143 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.448863 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.448978 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:19.453642 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:19.453654 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:19.453668 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:19.453706 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:19.457272 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:19.457878 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:19.457885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:19.457888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:19.458072 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:19.458467 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:19.458471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:19.458473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:19.458883 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:19.459166 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:19.459581 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 14:31:19.459825 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":498,"threads_started":1}
W20260810 14:31:19.461755 6569 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45096
Aug 10 14:31:19 dist-test-slave-q9h9 krb5kdc[6342](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:31:19 dist-test-slave-q9h9 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:31:19.508250 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:19.492714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45098 (local address 127.6.46.193:44779)
0810 14:31:19.492915 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:19.492923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:19.493341 (+ 418us) server_negotiation.cc:407] Connection header received
0810 14:31:19.493524 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:19.493529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:19.493625 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:19.493714 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:19.495595 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.496340 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.497978 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.498619 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.498738 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:19.502873 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:19.502878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:19.502882 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:19.502915 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:19.506167 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:19.506720 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:19.506723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:19.506725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:19.506841 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:19.507121 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:19.507124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:19.507126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:19.507347 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:19.507560 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:19.507757 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 14:31:19.507945 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:31:19.510111 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 6502
I20260810 14:31:19.524482 7171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:19.511083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.6.46.193:44779)
0810 14:31:19.511251 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:19.511256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:19.511804 (+ 548us) server_negotiation.cc:407] Connection header received
0810 14:31:19.511935 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:19.511938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:19.511990 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:19.512091 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:19.513476 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.514210 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.515838 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.516311 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.516404 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:19.519008 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:19.519027 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:19.519031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:19.519059 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:19.521974 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:19.522642 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:19.522647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:19.522651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:19.522782 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:19.523090 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:19.523093 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:19.523095 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:19.523403 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:19.523642 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:19.523823 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 14:31:19.523962 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:31:19.525139 6702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:19.607237 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 6502
I20260810 14:31:19.632781 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 6717
Aug 10 14:31:19 dist-test-slave-q9h9 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:31:19.648618 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:19.633820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56804 (local address 127.6.46.194:33211)
0810 14:31:19.634557 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:19.634564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:19.634591 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:31:19.634654 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:19.634660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:19.634893 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:19.635027 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:19.636155 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.637169 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.638486 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.639047 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.639140 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:19.643127 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:19.643138 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:19.643152 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:19.643184 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:19.646253 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:19.646859 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:19.646866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:19.646869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:19.646999 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:19.647348 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:19.647351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:19.647353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:19.647691 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:19.647926 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:19.648120 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 14:31:19.648313 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":431,"threads_started":1}
I20260810 14:31:19.649848 6914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:19.729652 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 6717
I20260810 14:31:19.753532 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 6930
Aug 10 14:31:19 dist-test-slave-q9h9 krb5kdc[6342](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:31:19.769290 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:19.754762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50824 (local address 127.6.46.195:32933)
0810 14:31:19.755687 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:19.755695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:19.755732 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:31:19.755815 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:19.755825 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:19.756126 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:19.756301 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:19.757654 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.758823 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.760014 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:19.760545 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:19.760636 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:19.764408 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:19.764418 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:19.764431 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:19.764462 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:19.767047 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:19.767655 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:19.767661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:19.767663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:19.767790 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:19.768054 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:19.768056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:19.768058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:19.768383 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:19.768605 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:19.768798 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 14:31:19.768986 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":737,"thread_start_us":552,"threads_started":1}
I20260810 14:31:19.770610 7146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:19.851079 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 6930
I20260810 14:31:19.871332 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 6362
I20260810 14:31:19.872531 6456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:19.973357 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 6362
2026-08-10T14:31:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4747 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:31:20 dist-test-slave-q9h9 krb5kdc[7188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:31:20 dist-test-slave-q9h9 krb5kdc[7188](info): set up 2 sockets
Aug 10 14:31:20 dist-test-slave-q9h9 krb5kdc[7188](info): commencing operation
krb5kdc: starting...
W20260810 14:31:21.166415 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:31:21 dist-test-slave-q9h9 krb5kdc[7188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:31:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:31:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:31:21.333788 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:45131
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:45131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:21.642740 7210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:21.643021 7210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:21.643158 7210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:21.650236 7210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:31:21.650336 7210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:21.650408 7210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:21.650482 7210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:31:21.650540 7210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:31:21.659982 7210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:45131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:45131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:21.661139 7210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:21.662357 7210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:21.673228 7224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:21.673211 7227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:21.673228 7230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:21.674266 7210 server_base.cc:1034] running on GCE node
I20260810 14:31:21.674986 7210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:21.676290 7210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:21.677489 7210 hybrid_clock.cc:648] HybridClock initialized: now 1786372281677449 us; error 24 us; skew 500 ppm
I20260810 14:31:21.677805 7210 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:21 dist-test-slave-q9h9 krb5kdc[7188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372281, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:21.680614 7210 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:31:21.681965 7210 webserver.cc:460] Webserver started at http://127.6.46.254:41419/ using document root <none> and password file <none>
I20260810 14:31:21.682626 7210 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:21.682744 7210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:21.683104 7210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:21.685865 7210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "84347b94ccd64ae4a3bfbd56934ee627"
format_stamp: "Formatted at 2026-08-10 14:31:21 on dist-test-slave-q9h9"
server_key: "606ad449d9731ccd4b4a332e180bd7f1"
server_key_iv: "0477e31a90e3e06c7d6e9e741ffdf981"
server_key_version: "encryptionkey@0"
I20260810 14:31:21.686611 7210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "84347b94ccd64ae4a3bfbd56934ee627"
format_stamp: "Formatted at 2026-08-10 14:31:21 on dist-test-slave-q9h9"
server_key: "606ad449d9731ccd4b4a332e180bd7f1"
server_key_iv: "0477e31a90e3e06c7d6e9e741ffdf981"
server_key_version: "encryptionkey@0"
I20260810 14:31:21.691303 7210 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:31:21.694502 7239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:21.695668 7210 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 14:31:21.695823 7210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "84347b94ccd64ae4a3bfbd56934ee627"
format_stamp: "Formatted at 2026-08-10 14:31:21 on dist-test-slave-q9h9"
server_key: "606ad449d9731ccd4b4a332e180bd7f1"
server_key_iv: "0477e31a90e3e06c7d6e9e741ffdf981"
server_key_version: "encryptionkey@0"
I20260810 14:31:21.695989 7210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:21.710754 7210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:21.714357 7210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:21.714600 7210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:21.731330 7210 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:45131
I20260810 14:31:21.731369 7315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:45131 every 8 connection(s)
I20260810 14:31:21.732662 7210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:31:21.736147 7317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:21.737210 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 7210
I20260810 14:31:21.737447 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:31:21.738277 6331 external_mini_cluster.cc:1428] Setting key 4a40fe63f35936e7616019043221fddb
I20260810 14:31:21.750046 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627: Bootstrap starting.
Aug 10 14:31:21 dist-test-slave-q9h9 krb5kdc[7188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:21.753670 7317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:21.754837 7317 log.cc:824] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:21.757766 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627: No bootstrap required, opened a new log
I20260810 14:31:21.760660 7321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:21.740294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.6.46.254:45131)
0810 14:31:21.741043 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:21.741065 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:21.741104 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:31:21.742690 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:21.742712 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:21.743087 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:21.743389 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:21.745093 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:21.746412 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:21.748122 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:21.748733 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:21.748853 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:21.753479 (+ 4626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:21.753487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:21.753500 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:21.753536 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:21.756764 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:21.757504 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:21.757510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:21.757516 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:21.757675 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:21.758094 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:21.758097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:21.758099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:21.758600 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:21.758822 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:21.759318 (+ 496us) server_negotiation.cc:299] Negotiation successful
0810 14:31:21.759654 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":362,"threads_started":1}
I20260810 14:31:21.766461 7317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84347b94ccd64ae4a3bfbd56934ee627" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45131 } }
I20260810 14:31:21.766780 7317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:21.766918 7317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84347b94ccd64ae4a3bfbd56934ee627, State: Initialized, Role: FOLLOWER
I20260810 14:31:21.767414 7317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84347b94ccd64ae4a3bfbd56934ee627" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45131 } }
I20260810 14:31:21.767620 7317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:31:21.767750 7317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:31:21.767892 7317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:21.769902 7317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84347b94ccd64ae4a3bfbd56934ee627" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45131 } }
I20260810 14:31:21.770371 7317 leader_election.cc:302] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84347b94ccd64ae4a3bfbd56934ee627; no voters:
I20260810 14:31:21.770876 7317 leader_election.cc:288] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:31:21.770974 7323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:21.771567 7323 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [term 1 LEADER]: Becoming Leader. State: Replica: 84347b94ccd64ae4a3bfbd56934ee627, State: Running, Role: LEADER
I20260810 14:31:21.772380 7323 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84347b94ccd64ae4a3bfbd56934ee627" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45131 } }
I20260810 14:31:21.772886 7317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:31:21.775403 7327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84347b94ccd64ae4a3bfbd56934ee627. Latest consensus state: current_term: 1 leader_uuid: "84347b94ccd64ae4a3bfbd56934ee627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84347b94ccd64ae4a3bfbd56934ee627" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45131 } } }
I20260810 14:31:21.775414 7325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84347b94ccd64ae4a3bfbd56934ee627" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84347b94ccd64ae4a3bfbd56934ee627" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45131 } } }
I20260810 14:31:21.775861 7325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:21.775861 7327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:21.782605 7333 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:31:21.787086 7333 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:31:21.795627 7333 catalog_manager.cc:1347] Generated new cluster ID: ab58f9b6af4a4e3ca0163aaf5f49edac
I20260810 14:31:21.795727 7333 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:31:21.827117 7333 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:31:21.828428 7333 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:31:21.837939 7333 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84347b94ccd64ae4a3bfbd56934ee627: Generated new TSK 0
I20260810 14:31:21.838738 7333 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:31:21.885094 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:45131
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:22.193660 7354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:22.193986 7354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:22.194146 7354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:22.201143 7354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:22.201236 7354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:22.201373 7354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:31:22.210625 7354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:45131
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:22.211786 7354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:22.213027 7354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:22.225116 7372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:22.225327 7363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:22.225170 7367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:22.226090 7354 server_base.cc:1034] running on GCE node
I20260810 14:31:22.226610 7354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:22.227407 7354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:22.228592 7354 hybrid_clock.cc:648] HybridClock initialized: now 1786372282228548 us; error 38 us; skew 500 ppm
I20260810 14:31:22.228842 7354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:22 dist-test-slave-q9h9 krb5kdc[7188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372282, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:22.231751 7354 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:31:22.233099 7354 webserver.cc:460] Webserver started at http://127.6.46.193:42879/ using document root <none> and password file <none>
I20260810 14:31:22.233718 7354 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:22.233821 7354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:22.234146 7354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:22.236829 7354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "98f2d72c481c4f0680ff44630d8569fa"
format_stamp: "Formatted at 2026-08-10 14:31:22 on dist-test-slave-q9h9"
server_key: "e74b58ca4cfbcf575461e0a2027784ac"
server_key_iv: "d16b392f8b82d33d6c15f2128b36f452"
server_key_version: "encryptionkey@0"
I20260810 14:31:22.237566 7354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "98f2d72c481c4f0680ff44630d8569fa"
format_stamp: "Formatted at 2026-08-10 14:31:22 on dist-test-slave-q9h9"
server_key: "e74b58ca4cfbcf575461e0a2027784ac"
server_key_iv: "d16b392f8b82d33d6c15f2128b36f452"
server_key_version: "encryptionkey@0"
I20260810 14:31:22.242502 7354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260810 14:31:22.245867 7387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:22.247169 7354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:31:22.247298 7354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "98f2d72c481c4f0680ff44630d8569fa"
format_stamp: "Formatted at 2026-08-10 14:31:22 on dist-test-slave-q9h9"
server_key: "e74b58ca4cfbcf575461e0a2027784ac"
server_key_iv: "d16b392f8b82d33d6c15f2128b36f452"
server_key_version: "encryptionkey@0"
I20260810 14:31:22.247452 7354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:22.271190 7354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:22.275133 7354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:22.275369 7354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:22.276261 7354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:22.277500 7354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:22.277573 7354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:22.277661 7354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:22.277704 7354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:22.308403 7354 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:40583
I20260810 14:31:22.308447 7580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:40583 every 8 connection(s)
I20260810 14:31:22.309654 7354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:31:22.320086 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 7354
I20260810 14:31:22.320339 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:31:22.321130 6331 external_mini_cluster.cc:1428] Setting key cd6172e066d1e57d7e4bca88285dae86
Aug 10 14:31:22 dist-test-slave-q9h9 krb5kdc[7188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372282, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:22.328724 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:22.312033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:43057 (local address 127.6.46.254:45131)
0810 14:31:22.312512 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:22.312519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:22.313404 (+ 885us) server_negotiation.cc:407] Connection header received
0810 14:31:22.315177 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:22.315182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:22.315250 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:22.315348 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:22.316764 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:22.317508 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:22.319034 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:22.319559 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:22.319665 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:22.323492 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:22.323497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:22.323500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:22.323530 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:22.325900 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:22.326590 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:22.326594 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:22.326597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:22.326715 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:22.327120 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:22.327124 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:22.327126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:22.327391 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:22.327520 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:22.328168 (+ 648us) server_negotiation.cc:299] Negotiation successful
0810 14:31:22.328384 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":223,"threads_started":1}
I20260810 14:31:22.329540 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:22.312506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:45131 (local address 127.6.46.193:43057)
0810 14:31:22.313242 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:22.313283 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:22.314923 (+ 1640us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:22.315529 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:22.315537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:22.316070 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:22.316604 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:22.316621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:22.317657 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:22.317661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:22.318907 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:22.318920 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:22.319815 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:22.319825 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:22.320142 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:22.320819 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:22.320857 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:22.323294 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:22.326045 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:22.326054 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:22.326069 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:22.326466 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:22.326818 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:22.326823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:22.326827 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:22.327002 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:22.327513 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:22.327520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:22.327823 (+ 303us) client_negotiation.cc:769] Sending connection context
0810 14:31:22.328044 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 14:31:22.328522 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":285,"threads_started":1}
I20260810 14:31:22.330935 7581 heartbeater.cc:347] Connected to a master server at 127.6.46.254:45131
I20260810 14:31:22.331212 7581 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:22.331856 7581 heartbeater.cc:511] Master 127.6.46.254:45131 requested a full tablet report, sending...
I20260810 14:31:22.333758 7266 ts_manager.cc:194] Registered new tserver with Master: 98f2d72c481c4f0680ff44630d8569fa (127.6.46.193:40583)
I20260810 14:31:22.335217 7266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:43057
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:31:22.360932 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:45131
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:22.670560 7590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:22.670822 7590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:22.670948 7590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:22.678093 7590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:22.678188 7590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:22.678355 7590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:31:22.687836 7590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:45131
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:22.689002 7590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:22.690376 7590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:22.702796 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:22.702868 7602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:22.702847 7604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:22.703620 7590 server_base.cc:1034] running on GCE node
I20260810 14:31:22.704214 7590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:22.705106 7590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:22.706296 7590 hybrid_clock.cc:648] HybridClock initialized: now 1786372282706248 us; error 34 us; skew 500 ppm
I20260810 14:31:22.706590 7590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:22 dist-test-slave-q9h9 krb5kdc[7188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372282, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:22.709432 7590 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:31:22.710825 7590 webserver.cc:460] Webserver started at http://127.6.46.194:45927/ using document root <none> and password file <none>
I20260810 14:31:22.711485 7590 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:22.711611 7590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:22.711901 7590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:22.714795 7590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "7f62c050a11e445a990d799dc5c654e4"
format_stamp: "Formatted at 2026-08-10 14:31:22 on dist-test-slave-q9h9"
server_key: "d01a27bb340b00dfae795ea74e2944d4"
server_key_iv: "b6b7c9f7f49e93bb38dc54a43f853a29"
server_key_version: "encryptionkey@0"
I20260810 14:31:22.715581 7590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "7f62c050a11e445a990d799dc5c654e4"
format_stamp: "Formatted at 2026-08-10 14:31:22 on dist-test-slave-q9h9"
server_key: "d01a27bb340b00dfae795ea74e2944d4"
server_key_iv: "b6b7c9f7f49e93bb38dc54a43f853a29"
server_key_version: "encryptionkey@0"
I20260810 14:31:22.720625 7590 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 14:31:22.723979 7620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:22.725234 7590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:31:22.725385 7590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "7f62c050a11e445a990d799dc5c654e4"
format_stamp: "Formatted at 2026-08-10 14:31:22 on dist-test-slave-q9h9"
server_key: "d01a27bb340b00dfae795ea74e2944d4"
server_key_iv: "b6b7c9f7f49e93bb38dc54a43f853a29"
server_key_version: "encryptionkey@0"
I20260810 14:31:22.725571 7590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:22.742875 7590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:22.746831 7590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:22.747076 7590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:22.747967 7590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:22.749219 7590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:22.749296 7590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:22.749416 7590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:22.749485 7590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:22.784228 7590 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:32897
I20260810 14:31:22.784271 7825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:32897 every 8 connection(s)
I20260810 14:31:22.785437 7590 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:31:22.786286 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 7590
I20260810 14:31:22.786484 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:31:22.787006 6331 external_mini_cluster.cc:1428] Setting key fa300d911e212af58453748d64036efe
Aug 10 14:31:22 dist-test-slave-q9h9 krb5kdc[7188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372282, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
I20260810 14:31:22.805260 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:22.788127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:37847 (local address 127.6.46.254:45131)
0810 14:31:22.788311 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:22.788317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:22.789615 (+ 1298us) server_negotiation.cc:407] Connection header received
0810 14:31:22.791455 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:22.791460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:22.791542 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:22.791645 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:22.793081 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:22.793983 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:22.795520 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:22.796102 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:22.796206 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:22.799947 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:22.799952 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:22.799955 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:22.799986 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:22.802503 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:22.803161 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:22.803166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:22.803170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:22.803297 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:22.803696 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:22.803700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:22.803703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:22.803951 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:22.804080 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:22.804731 (+ 651us) server_negotiation.cc:299] Negotiation successful
0810 14:31:22.804950 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:31:22.806222 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:22.788620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:45131 (local address 127.6.46.194:37847)
0810 14:31:22.789444 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:22.789499 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:22.791194 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:22.791840 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:22.791848 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:22.792375 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:22.792917 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:22.792934 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:22.794159 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:22.794164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:22.795377 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:22.795390 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:22.796316 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:22.796322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:22.796584 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:22.797254 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:22.797285 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:22.799741 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:22.802634 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:22.802643 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:22.802658 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:22.803043 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:22.803398 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:22.803403 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:22.803407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:22.803591 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:22.804080 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:22.804088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:22.804383 (+ 295us) client_negotiation.cc:769] Sending connection context
0810 14:31:22.804618 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 14:31:22.805108 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":357,"threads_started":1}
I20260810 14:31:22.807509 7829 heartbeater.cc:347] Connected to a master server at 127.6.46.254:45131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
I20260810 14:31:22.807799 7829 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:22.808633 7829 heartbeater.cc:511] Master 127.6.46.254:45131 requested a full tablet report, sending...
I20260810 14:31:22.810210 7266 ts_manager.cc:194] Registered new tserver with Master: 7f62c050a11e445a990d799dc5c654e4 (127.6.46.194:32897)
I20260810 14:31:22.811239 7266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:37847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:31:22.829761 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:45131
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:23.141398 7838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:23.141662 7838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:23.141790 7838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:23.148849 7838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:23.148952 7838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:23.149102 7838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:31:23.158635 7838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:45131
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:23.159803 7838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:23.161072 7838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:23.173247 7846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:23.173298 7848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:23.173319 7851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:23.174270 7838 server_base.cc:1034] running on GCE node
I20260810 14:31:23.174845 7838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:23.175683 7838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:23.176869 7838 hybrid_clock.cc:648] HybridClock initialized: now 1786372283176825 us; error 34 us; skew 500 ppm
I20260810 14:31:23.177162 7838 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:23 dist-test-slave-q9h9 krb5kdc[7188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:23.180152 7838 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:31:23.181548 7838 webserver.cc:460] Webserver started at http://127.6.46.195:38723/ using document root <none> and password file <none>
I20260810 14:31:23.182246 7838 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:23.182370 7838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:23.182637 7838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:23.185343 7838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5"
format_stamp: "Formatted at 2026-08-10 14:31:23 on dist-test-slave-q9h9"
server_key: "d7606bf91fd7dbb29c36e090ebe8cbab"
server_key_iv: "60cd098fb97b1351cf567512bfc00de1"
server_key_version: "encryptionkey@0"
I20260810 14:31:23.186167 7838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5"
format_stamp: "Formatted at 2026-08-10 14:31:23 on dist-test-slave-q9h9"
server_key: "d7606bf91fd7dbb29c36e090ebe8cbab"
server_key_iv: "60cd098fb97b1351cf567512bfc00de1"
server_key_version: "encryptionkey@0"
I20260810 14:31:23.190980 7838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:31:23.194172 7868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:23.195402 7838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:31:23.195531 7838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5"
format_stamp: "Formatted at 2026-08-10 14:31:23 on dist-test-slave-q9h9"
server_key: "d7606bf91fd7dbb29c36e090ebe8cbab"
server_key_iv: "60cd098fb97b1351cf567512bfc00de1"
server_key_version: "encryptionkey@0"
I20260810 14:31:23.195685 7838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:23.211180 7838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:23.214926 7838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:23.215154 7838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:23.215993 7838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:23.217243 7838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:23.217314 7838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:23.217406 7838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:23.217463 7838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:23.261152 7838 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:46169
I20260810 14:31:23.261240 8056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:46169 every 8 connection(s)
I20260810 14:31:23.262812 7838 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:31:23.265327 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 7838
I20260810 14:31:23.265640 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:31:23.266146 6331 external_mini_cluster.cc:1428] Setting key fd4a41d335fdf198b61ccabac1c2e181
Aug 10 14:31:23 dist-test-slave-q9h9 krb5kdc[7188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372283, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:23.284049 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.266252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:41601 (local address 127.6.46.254:45131)
0810 14:31:23.266453 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:23.266459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:23.267864 (+ 1405us) server_negotiation.cc:407] Connection header received
0810 14:31:23.269899 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:23.269903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:23.269972 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:23.270114 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:23.271585 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.272503 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.274142 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.274594 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.274703 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.278527 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:23.278532 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:23.278536 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:23.278580 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:23.281366 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.282119 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.282123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.282126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.282247 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.282612 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.282615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.282617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.282887 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:23.282992 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:23.283639 (+ 647us) server_negotiation.cc:299] Negotiation successful
0810 14:31:23.283795 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:31:23.284935 8060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.266776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:45131 (local address 127.6.46.195:41601)
0810 14:31:23.267683 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:23.267749 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:23.269660 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:23.270352 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:23.270359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:23.270864 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:23.271415 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:23.271431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.272670 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.272675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:23.273988 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:23.274001 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.274828 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.274836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:23.275072 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.275669 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:23.275691 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:23.278299 (+ 2608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:23.281496 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:23.281504 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:23.281518 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:23.281916 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:23.282333 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:23.282337 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:23.282339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:23.282522 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:23.282998 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:23.283004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:23.283310 (+ 306us) client_negotiation.cc:769] Sending connection context
0810 14:31:23.283519 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 14:31:23.283908 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":402,"threads_started":1}
I20260810 14:31:23.286262 8058 heartbeater.cc:347] Connected to a master server at 127.6.46.254:45131
I20260810 14:31:23.286587 8058 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:23.287249 8058 heartbeater.cc:511] Master 127.6.46.254:45131 requested a full tablet report, sending...
I20260810 14:31:23.288676 7266 ts_manager.cc:194] Registered new tserver with Master: 20840ec4ac554ffc9a3a9d29e8fb82b5 (127.6.46.195:46169)
I20260810 14:31:23.289492 7266 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:41601
I20260810 14:31:23.293356 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:31:23.315049 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.300968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.6.46.254:45131)
0810 14:31:23.301121 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:23.301125 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:23.301949 (+ 824us) server_negotiation.cc:407] Connection header received
0810 14:31:23.302173 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:23.302177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:23.302248 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:23.302332 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:23.304081 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.304775 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.306063 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.306606 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.306729 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.308985 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:23.308991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:23.308994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:23.309023 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:23.312235 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.312840 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.312844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.312846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.312976 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.313409 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.313415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.313418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.313739 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:23.313966 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:23.314463 (+ 497us) server_negotiation.cc:299] Negotiation successful
0810 14:31:23.314668 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:31:23.323551 7266 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:31:23.333741 7266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:31:23.337885 7581 heartbeater.cc:503] Master 127.6.46.254:45131 was elected leader, sending a full tablet report...
I20260810 14:31:23.377549 8078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.351060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:46169 (local address 127.0.0.1:37192)
0810 14:31:23.361237 (+ 10177us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:23.361285 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:23.361450 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:23.369173 (+ 7723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:23.369178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:23.369200 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:23.369865 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:23.369876 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.371422 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.371428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:23.374538 (+ 3110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:23.374550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.376511 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.376518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:23.376797 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.376838 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 14:31:23.376945 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 14:31:23.377176 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9935,"mutex_wait_us":32,"thread_start_us":12193,"threads_started":1}
I20260810 14:31:23.377538 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.351465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.6.46.195:46169)
0810 14:31:23.368255 (+ 16790us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:23.368264 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:23.368294 (+ 30us) server_negotiation.cc:407] Connection header received
0810 14:31:23.368385 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:23.368391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:23.368631 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:23.368870 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:23.370052 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.371245 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.374710 (+ 3465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.375924 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.376050 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.376277 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:23.377016 (+ 739us) server_negotiation.cc:299] Negotiation successful
0810 14:31:23.377186 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16613,"thread_start_us":17250,"threads_started":1}
I20260810 14:31:23.377897 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.351741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41828 (local address 127.6.46.194:32897)
0810 14:31:23.367141 (+ 15400us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:23.367150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:23.367183 (+ 33us) server_negotiation.cc:407] Connection header received
0810 14:31:23.367298 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:23.367306 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:23.367558 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:23.367758 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:23.368664 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.370692 (+ 2028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.374286 (+ 3594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.375911 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.376036 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.376219 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:23.377218 (+ 999us) server_negotiation.cc:299] Negotiation successful
0810 14:31:23.377361 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15233,"thread_start_us":15113,"threads_started":1}
I20260810 14:31:23.378305 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.352348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46790 (local address 127.6.46.193:40583)
0810 14:31:23.366690 (+ 14342us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:23.366699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:23.367695 (+ 996us) server_negotiation.cc:407] Connection header received
0810 14:31:23.367800 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:23.367808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:23.368039 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:23.368233 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:23.369205 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.370783 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.373842 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.375664 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.375806 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.376037 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:23.377115 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0810 14:31:23.377276 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14133,"thread_start_us":13951,"threads_started":1}
I20260810 14:31:23.378476 8077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.351213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:32897 (local address 127.0.0.1:41828)
0810 14:31:23.363495 (+ 12282us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:23.363513 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:23.363618 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:23.368023 (+ 4405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:23.368030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:23.368067 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:23.368466 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:23.368480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.370833 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.370839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:23.374144 (+ 3305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:23.374158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.376709 (+ 2551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.376717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:23.376914 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.376940 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:31:23.377004 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 14:31:23.378124 (+ 1120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12148,"mutex_wait_us":188,"thread_start_us":9648,"threads_started":1}
I20260810 14:31:23.378912 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.351668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:40583 (local address 127.0.0.1:46790)
0810 14:31:23.367535 (+ 15867us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:23.367552 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:23.367675 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:23.368496 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:23.368501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:23.368531 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:23.369001 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:23.369014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.370926 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.370935 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:23.373663 (+ 2728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:23.373677 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.376496 (+ 2819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.376505 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:23.376829 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.376915 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 14:31:23.377003 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 14:31:23.377772 (+ 769us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15759,"mutex_wait_us":738,"thread_start_us":15628,"threads_started":1}
I20260810 14:31:23.382483 7701 tablet_service.cc:1467] Processing CreateTablet for tablet 429cc232e2e149c687d0141e2ced4d4e (DEFAULT_TABLE table=test-table [id=a4161a416bf24892913b8d0f1fc3c6a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:23.382550 7473 tablet_service.cc:1467] Processing CreateTablet for tablet 429cc232e2e149c687d0141e2ced4d4e (DEFAULT_TABLE table=test-table [id=a4161a416bf24892913b8d0f1fc3c6a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:23.383850 7701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 429cc232e2e149c687d0141e2ced4d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:23.384367 7473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 429cc232e2e149c687d0141e2ced4d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:23.383860 7941 tablet_service.cc:1467] Processing CreateTablet for tablet 429cc232e2e149c687d0141e2ced4d4e (DEFAULT_TABLE table=test-table [id=a4161a416bf24892913b8d0f1fc3c6a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:23.385478 7941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 429cc232e2e149c687d0141e2ced4d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:23.396914 8088 tablet_bootstrap.cc:492] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4: Bootstrap starting.
I20260810 14:31:23.400431 8088 tablet_bootstrap.cc:654] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:23.400951 8089 tablet_bootstrap.cc:492] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa: Bootstrap starting.
I20260810 14:31:23.402148 8090 tablet_bootstrap.cc:492] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5: Bootstrap starting.
I20260810 14:31:23.402892 8088 log.cc:824] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:23.405438 8088 tablet_bootstrap.cc:492] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4: No bootstrap required, opened a new log
I20260810 14:31:23.405579 8089 tablet_bootstrap.cc:654] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:23.405678 8088 ts_tablet_manager.cc:1397] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 14:31:23.406364 8090 tablet_bootstrap.cc:654] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:23.407474 8089 log.cc:824] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:23.408480 8090 log.cc:824] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:23.411046 8089 tablet_bootstrap.cc:492] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa: No bootstrap required, opened a new log
I20260810 14:31:23.411306 8089 ts_tablet_manager.cc:1397] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20260810 14:31:23.412744 8090 tablet_bootstrap.cc:492] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5: No bootstrap required, opened a new log
I20260810 14:31:23.413241 8090 ts_tablet_manager.cc:1397] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260810 14:31:23.418201 8088 raft_consensus.cc:348] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 46169 } } peers { permanent_uuid: "7f62c050a11e445a990d799dc5c654e4" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32897 } } peers { permanent_uuid: "98f2d72c481c4f0680ff44630d8569fa" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 40583 } }
I20260810 14:31:23.418705 8088 raft_consensus.cc:374] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:23.418901 8088 raft_consensus.cc:729] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f62c050a11e445a990d799dc5c654e4, State: Initialized, Role: FOLLOWER
I20260810 14:31:23.419672 8088 consensus_queue.cc:260] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 46169 } } peers { permanent_uuid: "7f62c050a11e445a990d799dc5c654e4" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32897 } } peers { permanent_uuid: "98f2d72c481c4f0680ff44630d8569fa" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 40583 } }
I20260810 14:31:23.420482 8089 raft_consensus.cc:348] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 46169 } } peers { permanent_uuid: "7f62c050a11e445a990d799dc5c654e4" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32897 } } peers { permanent_uuid: "98f2d72c481c4f0680ff44630d8569fa" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 40583 } }
I20260810 14:31:23.420955 8089 raft_consensus.cc:374] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:23.421065 8089 raft_consensus.cc:729] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f2d72c481c4f0680ff44630d8569fa, State: Initialized, Role: FOLLOWER
I20260810 14:31:23.421136 7829 heartbeater.cc:503] Master 127.6.46.254:45131 was elected leader, sending a full tablet report...
I20260810 14:31:23.421787 8088 ts_tablet_manager.cc:1428] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4: Time spent starting tablet: real 0.016s user 0.009s sys 0.008s
I20260810 14:31:23.421646 8089 consensus_queue.cc:260] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 46169 } } peers { permanent_uuid: "7f62c050a11e445a990d799dc5c654e4" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32897 } } peers { permanent_uuid: "98f2d72c481c4f0680ff44630d8569fa" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 40583 } }
I20260810 14:31:23.424526 8089 ts_tablet_manager.cc:1428] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa: Time spent starting tablet: real 0.013s user 0.003s sys 0.010s
I20260810 14:31:23.425319 8090 raft_consensus.cc:348] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 46169 } } peers { permanent_uuid: "7f62c050a11e445a990d799dc5c654e4" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32897 } } peers { permanent_uuid: "98f2d72c481c4f0680ff44630d8569fa" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 40583 } }
I20260810 14:31:23.425706 8090 raft_consensus.cc:374] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:23.425822 8090 raft_consensus.cc:729] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20840ec4ac554ffc9a3a9d29e8fb82b5, State: Initialized, Role: FOLLOWER
I20260810 14:31:23.426388 8090 consensus_queue.cc:260] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 46169 } } peers { permanent_uuid: "7f62c050a11e445a990d799dc5c654e4" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32897 } } peers { permanent_uuid: "98f2d72c481c4f0680ff44630d8569fa" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 40583 } }
I20260810 14:31:23.428761 8058 heartbeater.cc:503] Master 127.6.46.254:45131 was elected leader, sending a full tablet report...
I20260810 14:31:23.429296 8090 ts_tablet_manager.cc:1428] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260810 14:31:23.514804 8059 tablet.cc:2369] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:31:23.537372 7830 tablet.cc:2369] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:31:23.542034 8104 raft_consensus.cc:482] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:31:23.542325 8104 raft_consensus.cc:504] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 46169 } } peers { permanent_uuid: "7f62c050a11e445a990d799dc5c654e4" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32897 } } peers { permanent_uuid: "98f2d72c481c4f0680ff44630d8569fa" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 40583 } }
I20260810 14:31:23.544486 8104 leader_election.cc:288] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20840ec4ac554ffc9a3a9d29e8fb82b5 (127.6.46.195:46169), 7f62c050a11e445a990d799dc5c654e4 (127.6.46.194:32897)
I20260810 14:31:23.551401 8106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.544799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:32897 (local address 127.6.46.193:38313)
0810 14:31:23.545551 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:23.545566 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:23.545722 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:23.546062 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:23.546067 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:23.546094 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:23.546745 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:23.546756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.547718 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.547722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:23.549449 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:23.549463 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.550738 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.550745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:23.550861 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.550885 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:31:23.550974 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 14:31:23.551067 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":1,"thread_start_us":483,"threads_started":1}
I20260810 14:31:23.551452 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.544959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:38313 (local address 127.6.46.194:32897)
0810 14:31:23.545147 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:23.545152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:23.545657 (+ 505us) server_negotiation.cc:407] Connection header received
0810 14:31:23.545867 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:23.545871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:23.545932 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:23.546006 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:23.546885 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.547596 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.549615 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.550455 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.550619 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.550699 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:23.551044 (+ 345us) server_negotiation.cc:299] Negotiation successful
0810 14:31:23.551196 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:31:23.552603 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.544848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:57385 (local address 127.6.46.195:46169)
0810 14:31:23.545072 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:23.545078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:23.546301 (+ 1223us) server_negotiation.cc:407] Connection header received
0810 14:31:23.546526 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:23.546530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:23.546589 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:23.546665 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:23.547278 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.548205 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.550477 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.551784 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.551938 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.552008 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:23.552334 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 14:31:23.552432 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 14:31:23.552802 7738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "429cc232e2e149c687d0141e2ced4d4e" candidate_uuid: "98f2d72c481c4f0680ff44630d8569fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f62c050a11e445a990d799dc5c654e4" is_pre_election: true
I20260810 14:31:23.553196 8107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.544839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:46169 (local address 127.6.46.193:57385)
0810 14:31:23.546195 (+ 1356us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:23.546214 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:23.546382 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:23.546712 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:23.546716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:23.546744 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:23.547082 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:23.547091 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.548375 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.548379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:23.550327 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:23.550339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.551989 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.551995 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:23.552151 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.552184 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 14:31:23.552696 (+ 512us) client_negotiation.cc:240] Negotiation successful
0810 14:31:23.552823 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1150,"mutex_wait_us":20,"thread_start_us":860,"threads_started":1}
I20260810 14:31:23.553642 7738 raft_consensus.cc:2454] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f2d72c481c4f0680ff44630d8569fa in term 0.
I20260810 14:31:23.554423 7393 leader_election.cc:302] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f62c050a11e445a990d799dc5c654e4, 98f2d72c481c4f0680ff44630d8569fa; no voters:
I20260810 14:31:23.554795 8104 raft_consensus.cc:2781] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:31:23.554917 8104 raft_consensus.cc:482] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:31:23.555008 8104 raft_consensus.cc:3037] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:23.554934 7989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "429cc232e2e149c687d0141e2ced4d4e" candidate_uuid: "98f2d72c481c4f0680ff44630d8569fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5" is_pre_election: true
I20260810 14:31:23.555606 7989 raft_consensus.cc:2454] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f2d72c481c4f0680ff44630d8569fa in term 0.
I20260810 14:31:23.557646 8104 raft_consensus.cc:504] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 46169 } } peers { permanent_uuid: "7f62c050a11e445a990d799dc5c654e4" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32897 } } peers { permanent_uuid: "98f2d72c481c4f0680ff44630d8569fa" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 40583 } }
I20260810 14:31:23.558485 8104 leader_election.cc:288] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [CANDIDATE]: Term 1 election: Requested vote from peers 20840ec4ac554ffc9a3a9d29e8fb82b5 (127.6.46.195:46169), 7f62c050a11e445a990d799dc5c654e4 (127.6.46.194:32897)
I20260810 14:31:23.558807 7989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "429cc232e2e149c687d0141e2ced4d4e" candidate_uuid: "98f2d72c481c4f0680ff44630d8569fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5"
I20260810 14:31:23.559015 7989 raft_consensus.cc:3037] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:23.559000 7738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "429cc232e2e149c687d0141e2ced4d4e" candidate_uuid: "98f2d72c481c4f0680ff44630d8569fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f62c050a11e445a990d799dc5c654e4"
I20260810 14:31:23.559276 7738 raft_consensus.cc:3037] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4 [term 0 FOLLOWER]: Advancing to term 1
W20260810 14:31:23.561822 7582 tablet.cc:2369] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:31:23.562505 7738 raft_consensus.cc:2454] T 429cc232e2e149c687d0141e2ced4d4e P 7f62c050a11e445a990d799dc5c654e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f2d72c481c4f0680ff44630d8569fa in term 1.
I20260810 14:31:23.562510 7989 raft_consensus.cc:2454] T 429cc232e2e149c687d0141e2ced4d4e P 20840ec4ac554ffc9a3a9d29e8fb82b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f2d72c481c4f0680ff44630d8569fa in term 1.
I20260810 14:31:23.562872 7393 leader_election.cc:302] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f62c050a11e445a990d799dc5c654e4, 98f2d72c481c4f0680ff44630d8569fa; no voters:
I20260810 14:31:23.563151 8104 raft_consensus.cc:2781] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:23.563700 8104 raft_consensus.cc:686] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [term 1 LEADER]: Becoming Leader. State: Replica: 98f2d72c481c4f0680ff44630d8569fa, State: Running, Role: LEADER
I20260810 14:31:23.564174 8104 consensus_queue.cc:237] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 46169 } } peers { permanent_uuid: "7f62c050a11e445a990d799dc5c654e4" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32897 } } peers { permanent_uuid: "98f2d72c481c4f0680ff44630d8569fa" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 40583 } }
I20260810 14:31:23.569154 7266 catalog_manager.cc:5495] T 429cc232e2e149c687d0141e2ced4d4e P 98f2d72c481c4f0680ff44630d8569fa reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f2d72c481c4f0680ff44630d8569fa (127.6.46.193). New cstate: current_term: 1 leader_uuid: "98f2d72c481c4f0680ff44630d8569fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20840ec4ac554ffc9a3a9d29e8fb82b5" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 46169 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f62c050a11e445a990d799dc5c654e4" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f2d72c481c4f0680ff44630d8569fa" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 40583 } health_report { overall_health: HEALTHY } } }
Aug 10 14:31:23 dist-test-slave-q9h9 krb5kdc[7188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:31:23.615751 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.601075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46802 (local address 127.6.46.193:40583)
0810 14:31:23.601250 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:23.601261 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:23.601878 (+ 617us) server_negotiation.cc:407] Connection header received
0810 14:31:23.602109 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:23.602114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:23.602198 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:23.602281 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:23.604019 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.604720 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.605961 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.606401 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.606509 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.610578 (+ 4069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:23.610590 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:23.610606 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:23.610652 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:23.613159 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.613757 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.613764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.613767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.613958 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.614324 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.614328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.614330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.614650 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:23.614930 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:23.615213 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 14:31:23.615454 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 14:31:23 dist-test-slave-q9h9 krb5kdc[7188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372283, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:31:23 dist-test-slave-q9h9 krb5kdc[7188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372283, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:31:23.666579 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.652526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46804 (local address 127.6.46.193:40583)
0810 14:31:23.652749 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:23.652756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:23.653026 (+ 270us) server_negotiation.cc:407] Connection header received
0810 14:31:23.653211 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:23.653216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:23.653281 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:23.653365 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:23.655356 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.656073 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.657433 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.657932 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.658068 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.661672 (+ 3604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:23.661677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:23.661681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:23.661714 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:23.664306 (+ 2592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.664839 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.664843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.664845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.664971 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.665267 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.665270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.665272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.665554 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:23.665889 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:23.666010 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 14:31:23.666234 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 14:31:23 dist-test-slave-q9h9 krb5kdc[7188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:31:23 dist-test-slave-q9h9 krb5kdc[7188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:31:23.712345 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.697434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.6.46.193:40583)
0810 14:31:23.697632 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:23.697639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:23.697907 (+ 268us) server_negotiation.cc:407] Connection header received
0810 14:31:23.698097 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:23.698101 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:23.698183 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:23.698269 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:23.700614 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.701367 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.702752 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.703293 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.703416 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.707450 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:23.707455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:23.707458 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:23.707490 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:23.710301 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.710859 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.710862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.710865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.710990 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.711248 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.711250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.711253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.711497 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:23.711695 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:23.711888 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 14:31:23.712076 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 14:31:23.714506 7441 tablet_service.cc:2908] Rejecting scan request for tablet 429cc232e2e149c687d0141e2ced4d4e: Uninitialized: clean time has not yet been initialized
I20260810 14:31:23.718452 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 7354
I20260810 14:31:23.730412 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.719243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46816 (local address 127.6.46.193:40583)
0810 14:31:23.719442 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:23.719448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:23.719764 (+ 316us) server_negotiation.cc:407] Connection header received
0810 14:31:23.719883 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:23.719886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:23.719941 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:23.720011 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:23.721103 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.721828 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.723019 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.723423 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.723512 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.725393 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:23.725398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:23.725401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:23.725429 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:23.728135 (+ 2706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.728738 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.728742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.728744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.728881 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.729166 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.729169 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.729172 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.729432 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:23.729631 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:23.729862 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 14:31:23.730106 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 14:31:23.731065 7575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:23.823793 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 7354
I20260810 14:31:23.848928 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 7590
Aug 10 14:31:23 dist-test-slave-q9h9 krb5kdc[7188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:31:23.863736 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.849677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.6.46.194:32897)
0810 14:31:23.849864 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:23.849870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:23.849891 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:31:23.850116 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:23.850120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:23.850188 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:23.850283 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:23.851817 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.852562 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.853986 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.854467 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.854583 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.858488 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:23.858506 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:23.858525 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:23.858569 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:23.861194 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.861778 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.861785 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.861787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.861908 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.862449 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.862452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.862454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.862853 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:23.863095 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:23.863310 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 14:31:23.863472 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 14:31:23.864375 7814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:23.950212 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 7590
I20260810 14:31:23.972306 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 7838
Aug 10 14:31:23 dist-test-slave-q9h9 krb5kdc[7188](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372283, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:31:23.986439 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:23.973106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.6.46.195:46169)
0810 14:31:23.973328 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:23.973332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:23.973351 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:31:23.973414 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:23.973418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:23.973482 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:23.973577 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:23.974828 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.975541 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.976852 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:23.977238 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:23.977337 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:23.981447 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:23.981465 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:23.981483 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:23.981519 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:23.984106 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.984686 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.984692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.984695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.984842 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:23.985120 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:23.985123 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:23.985125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:23.985467 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:23.985680 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:23.985876 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 14:31:23.986086 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 14:31:23.987116 8050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:24.072377 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 7838
I20260810 14:31:24.093986 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 7210
I20260810 14:31:24.095217 7311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:24.201135 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 7210
2026-08-10T14:31:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4230 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:31:24 dist-test-slave-q9h9 krb5kdc[8145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:31:24 dist-test-slave-q9h9 krb5kdc[8145](info): set up 2 sockets
Aug 10 14:31:24 dist-test-slave-q9h9 krb5kdc[8145](info): commencing operation
krb5kdc: starting...
W20260810 14:31:25.411670 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.163s user 0.005s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:31:25 dist-test-slave-q9h9 krb5kdc[8145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:31:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:31:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:31:25.591065 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:41019
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:41019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:25.905318 8166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:25.905601 8166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:25.905730 8166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:25.912693 8166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:31:25.912794 8166 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 14:31:25.912861 8166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:25.912945 8166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:25.913007 8166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:31:25.913087 8166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:31:25.922576 8166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:41019
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:41019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:25.923815 8166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:25.925081 8166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:25.936283 8178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:25.936322 8182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:25.936357 8180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:25.937399 8166 server_base.cc:1034] running on GCE node
I20260810 14:31:25.938220 8166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:25.939616 8166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:25.940838 8166 hybrid_clock.cc:648] HybridClock initialized: now 1786372285940765 us; error 49 us; skew 500 ppm
I20260810 14:31:25.941144 8166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:25 dist-test-slave-q9h9 krb5kdc[8145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372285, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:25.944216 8166 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:31:25.945669 8166 webserver.cc:460] Webserver started at http://127.6.46.254:34419/ using document root <none> and password file <none>
I20260810 14:31:25.946379 8166 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:25.946468 8166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:25.946736 8166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:25.949570 8166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "dfc4b30d653e47ebaf0d3ae1f8ce8c70"
format_stamp: "Formatted at 2026-08-10 14:31:25 on dist-test-slave-q9h9"
server_key: "454bda1a07bdd94e2768a2c78bb44593"
server_key_iv: "53bdf62616959da2f82cc2dd5ca85cf8"
server_key_version: "encryptionkey@0"
I20260810 14:31:25.950379 8166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "dfc4b30d653e47ebaf0d3ae1f8ce8c70"
format_stamp: "Formatted at 2026-08-10 14:31:25 on dist-test-slave-q9h9"
server_key: "454bda1a07bdd94e2768a2c78bb44593"
server_key_iv: "53bdf62616959da2f82cc2dd5ca85cf8"
server_key_version: "encryptionkey@0"
I20260810 14:31:25.955595 8166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 14:31:25.959002 8193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:25.960278 8166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:31:25.960438 8166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "dfc4b30d653e47ebaf0d3ae1f8ce8c70"
format_stamp: "Formatted at 2026-08-10 14:31:25 on dist-test-slave-q9h9"
server_key: "454bda1a07bdd94e2768a2c78bb44593"
server_key_iv: "53bdf62616959da2f82cc2dd5ca85cf8"
server_key_version: "encryptionkey@0"
I20260810 14:31:25.960616 8166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:25.980741 8166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:25.984872 8166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:25.985081 8166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:26.004572 8166 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:41019
I20260810 14:31:26.004597 8304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:41019 every 8 connection(s)
I20260810 14:31:26.005897 8166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:31:26.009486 8305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:26.016238 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 8166
I20260810 14:31:26.016446 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:31:26.017082 6331 external_mini_cluster.cc:1428] Setting key 6f61f0302d97f3640d4288eda19e6fb9
I20260810 14:31:26.019356 8305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70: Bootstrap starting.
I20260810 14:31:26.022446 8305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:26.023660 8305 log.cc:824] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:26.026561 8305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70: No bootstrap required, opened a new log
Aug 10 14:31:26 dist-test-slave-q9h9 krb5kdc[8145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:26.036007 8305 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc4b30d653e47ebaf0d3ae1f8ce8c70" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41019 } }
I20260810 14:31:26.036374 8305 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:26.036466 8305 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc4b30d653e47ebaf0d3ae1f8ce8c70, State: Initialized, Role: FOLLOWER
I20260810 14:31:26.037103 8305 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc4b30d653e47ebaf0d3ae1f8ce8c70" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41019 } }
I20260810 14:31:26.037374 8305 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:31:26.037516 8305 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:31:26.037712 8305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:26.040319 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:26.018919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.6.46.254:41019)
0810 14:31:26.019678 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:26.019690 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:26.019729 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:31:26.021384 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:26.021405 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:26.021742 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:26.022073 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:26.023780 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:26.025118 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:26.026784 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:26.027462 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:26.027593 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:26.032075 (+ 4482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:26.032086 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:26.032104 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:26.032156 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:26.036025 (+ 3869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:26.036734 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:26.036741 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:26.036749 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:26.036943 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:26.037299 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:26.037303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:26.037306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:26.037977 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:26.038315 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:26.038615 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 14:31:26.039033 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":343,"threads_started":1}
I20260810 14:31:26.040853 8305 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc4b30d653e47ebaf0d3ae1f8ce8c70" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41019 } }
I20260810 14:31:26.041303 8305 leader_election.cc:302] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfc4b30d653e47ebaf0d3ae1f8ce8c70; no voters:
I20260810 14:31:26.041852 8305 leader_election.cc:288] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:31:26.041992 8314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:26.043383 8314 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [term 1 LEADER]: Becoming Leader. State: Replica: dfc4b30d653e47ebaf0d3ae1f8ce8c70, State: Running, Role: LEADER
I20260810 14:31:26.043983 8305 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:31:26.043910 8314 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc4b30d653e47ebaf0d3ae1f8ce8c70" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41019 } }
I20260810 14:31:26.046557 8315 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfc4b30d653e47ebaf0d3ae1f8ce8c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc4b30d653e47ebaf0d3ae1f8ce8c70" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41019 } } }
I20260810 14:31:26.046562 8316 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfc4b30d653e47ebaf0d3ae1f8ce8c70. Latest consensus state: current_term: 1 leader_uuid: "dfc4b30d653e47ebaf0d3ae1f8ce8c70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfc4b30d653e47ebaf0d3ae1f8ce8c70" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41019 } } }
I20260810 14:31:26.046985 8316 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:26.046985 8315 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:26.047809 8329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:31:26.053035 8329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:31:26.062415 8329 catalog_manager.cc:1347] Generated new cluster ID: c1559655778a42d6847cf7b8fc87bcaa
I20260810 14:31:26.062515 8329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:31:26.076610 8329 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:31:26.077773 8329 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:31:26.085646 8329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70: Generated new TSK 0
I20260810 14:31:26.086485 8329 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:31:26.138384 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:41019
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:26.451812 8348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:26.452091 8348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:26.452236 8348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:26.459324 8348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:26.459437 8348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:26.459555 8348 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:31:26.459669 8348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:31:26.468974 8348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.46.254:41019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:26.470178 8348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:26.471458 8348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:26.483564 8354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:26.483664 8360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:26.483629 8356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:26.484395 8348 server_base.cc:1034] running on GCE node
I20260810 14:31:26.484997 8348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:26.485850 8348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:26.487082 8348 hybrid_clock.cc:648] HybridClock initialized: now 1786372286487010 us; error 43 us; skew 500 ppm
I20260810 14:31:26.487393 8348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:26 dist-test-slave-q9h9 krb5kdc[8145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:26.490397 8348 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:31:26.491694 8348 webserver.cc:460] Webserver started at http://127.6.46.193:46495/ using document root <none> and password file <none>
I20260810 14:31:26.492328 8348 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:26.492429 8348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:26.492725 8348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:26.495479 8348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "87e17816a9ba4658ab9cc26ccc56894f"
format_stamp: "Formatted at 2026-08-10 14:31:26 on dist-test-slave-q9h9"
server_key: "7eb99ca6da4729058cc9ab558eb3eec6"
server_key_iv: "6520cedaac2ad82e17251b336eb5221d"
server_key_version: "encryptionkey@0"
I20260810 14:31:26.496240 8348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "87e17816a9ba4658ab9cc26ccc56894f"
format_stamp: "Formatted at 2026-08-10 14:31:26 on dist-test-slave-q9h9"
server_key: "7eb99ca6da4729058cc9ab558eb3eec6"
server_key_iv: "6520cedaac2ad82e17251b336eb5221d"
server_key_version: "encryptionkey@0"
I20260810 14:31:26.501107 8348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 14:31:26.504454 8375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:26.505682 8348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:31:26.505844 8348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "87e17816a9ba4658ab9cc26ccc56894f"
format_stamp: "Formatted at 2026-08-10 14:31:26 on dist-test-slave-q9h9"
server_key: "7eb99ca6da4729058cc9ab558eb3eec6"
server_key_iv: "6520cedaac2ad82e17251b336eb5221d"
server_key_version: "encryptionkey@0"
I20260810 14:31:26.506065 8348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:26.517719 8348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:26.521274 8348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:26.521507 8348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:26.523923 8348 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:26.524008 8348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:26.524153 8348 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:26.524219 8348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:31:26 dist-test-slave-q9h9 krb5kdc[8145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:26.546207 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:26.524625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:34799 (local address 127.6.46.254:41019)
0810 14:31:26.524813 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:26.524820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:26.526252 (+ 1432us) server_negotiation.cc:407] Connection header received
0810 14:31:26.528184 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:26.528190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:26.528279 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:26.528376 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:26.530907 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:26.531996 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:26.534154 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:26.534694 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:26.534829 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:26.539849 (+ 5020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:26.539856 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:26.539860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:26.539892 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:26.543043 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:26.543755 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:26.543760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:26.543764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:26.543923 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:26.544310 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:26.544316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:26.544319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:26.544691 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:26.544859 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:26.545498 (+ 639us) server_negotiation.cc:299] Negotiation successful
0810 14:31:26.545787 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 14:31:26.546998 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:26.525163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41019 (local address 127.6.46.193:34799)
0810 14:31:26.526069 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:26.526110 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:26.527912 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:26.528792 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:26.528804 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:26.529650 (+ 846us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:26.530654 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:26.530686 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:26.532147 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:26.532151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:26.533968 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:26.533994 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:26.534981 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:26.534989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:26.535287 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:26.536054 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:26.536088 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:26.539571 (+ 3483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:26.543188 (+ 3617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:26.543197 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:26.543211 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:26.543641 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:26.544029 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:26.544033 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:26.544035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:26.544215 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:26.544827 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:26.544833 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:26.545166 (+ 333us) client_negotiation.cc:769] Sending connection context
0810 14:31:26.545376 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 14:31:26.545848 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":404,"threads_started":1}
I20260810 14:31:26.566174 8348 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:46501
I20260810 14:31:26.566215 8576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:46501 every 8 connection(s)
I20260810 14:31:26.567394 8348 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:31:26.567518 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:26.553472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:54243 (local address 127.6.46.254:41019)
0810 14:31:26.553656 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:26.553663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:26.554700 (+ 1037us) server_negotiation.cc:407] Connection header received
0810 14:31:26.554881 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:26.554886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:26.554966 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:26.555054 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:26.556468 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:26.557420 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:26.558979 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:26.559526 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:26.559638 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:26.561983 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:26.561988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:26.561993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:26.562064 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:26.564858 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:26.565574 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:26.565579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:26.565583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:26.565725 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:26.566096 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:26.566102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:26.566106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:26.566424 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:26.566566 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:26.566956 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 14:31:26.567170 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 14:31:26.567711 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:26.553469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41019 (local address 127.6.46.193:54243)
0810 14:31:26.554591 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:26.554614 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:26.554753 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:26.555209 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:26.555212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:26.555824 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:26.556311 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:26.556325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:26.557555 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:26.557561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:26.558841 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:26.558855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:26.559920 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:26.559930 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:26.560072 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:26.560719 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:26.560743 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:26.561767 (+ 1024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:26.564961 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:26.564966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:26.564970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:26.565470 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:26.565834 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:26.565839 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:26.565843 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:26.565976 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:26.566537 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:26.566542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:26.566718 (+ 176us) client_negotiation.cc:769] Sending connection context
0810 14:31:26.566831 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 14:31:26.567183 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":449,"threads_started":1}
I20260810 14:31:26.568203 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 8348
I20260810 14:31:26.568415 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:31:26.569212 6331 external_mini_cluster.cc:1428] Setting key 5493b68cf06d032fa6e3817fa499c4ec
I20260810 14:31:26.569561 8577 heartbeater.cc:347] Connected to a master server at 127.6.46.254:41019
I20260810 14:31:26.569810 8577 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:26.570729 8577 heartbeater.cc:511] Master 127.6.46.254:41019 requested a full tablet report, sending...
I20260810 14:31:26.573711 8227 ts_manager.cc:194] Registered new tserver with Master: 87e17816a9ba4658ab9cc26ccc56894f (127.6.46.193:46501)
I20260810 14:31:26.575572 8227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:34799
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:31:26.611241 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:41019
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:26.921005 8584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:26.921262 8584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:26.921383 8584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:26.928328 8584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:26.928422 8584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:26.928529 8584 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:31:26.928620 8584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:31:26.938077 8584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.46.254:41019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:26.939240 8584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:26.940553 8584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:26.952802 8595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:26.952967 8598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:26.952850 8594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:26.953914 8584 server_base.cc:1034] running on GCE node
I20260810 14:31:26.954507 8584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:26.955395 8584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:26.956620 8584 hybrid_clock.cc:648] HybridClock initialized: now 1786372286956549 us; error 38 us; skew 500 ppm
I20260810 14:31:26.956902 8584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:26 dist-test-slave-q9h9 krb5kdc[8145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:26.959971 8584 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:31:26.961449 8584 webserver.cc:460] Webserver started at http://127.6.46.194:34861/ using document root <none> and password file <none>
I20260810 14:31:26.962179 8584 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:26.962297 8584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:26.962601 8584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:26.965466 8584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "99e80f721f8a470d8e20ad5855510c7d"
format_stamp: "Formatted at 2026-08-10 14:31:26 on dist-test-slave-q9h9"
server_key: "056207670b12a890a928fbf25b99e5e7"
server_key_iv: "1c3198be15c2ae95e984d1ee23778524"
server_key_version: "encryptionkey@0"
I20260810 14:31:26.966256 8584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "99e80f721f8a470d8e20ad5855510c7d"
format_stamp: "Formatted at 2026-08-10 14:31:26 on dist-test-slave-q9h9"
server_key: "056207670b12a890a928fbf25b99e5e7"
server_key_iv: "1c3198be15c2ae95e984d1ee23778524"
server_key_version: "encryptionkey@0"
I20260810 14:31:26.971038 8584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:31:26.974246 8608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:26.975457 8584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:31:26.975601 8584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "99e80f721f8a470d8e20ad5855510c7d"
format_stamp: "Formatted at 2026-08-10 14:31:26 on dist-test-slave-q9h9"
server_key: "056207670b12a890a928fbf25b99e5e7"
server_key_iv: "1c3198be15c2ae95e984d1ee23778524"
server_key_version: "encryptionkey@0"
I20260810 14:31:26.975796 8584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:26.993146 8584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:26.996999 8584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:26.997237 8584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:26.999856 8584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:26.999938 8584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:27.000123 8584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:27.000247 8584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:31:27 dist-test-slave-q9h9 krb5kdc[8145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372286, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:27.023983 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.000243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:48481 (local address 127.6.46.254:41019)
0810 14:31:27.000441 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:27.000447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:27.001752 (+ 1305us) server_negotiation.cc:407] Connection header received
0810 14:31:27.003782 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:27.003787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:27.003871 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:27.003955 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:27.006941 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.008199 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.010940 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.011634 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.011737 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.016729 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:27.016736 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:27.016741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:27.016779 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:27.020350 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:27.021327 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:27.021332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:27.021336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:27.021463 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:27.022003 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:27.022008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:27.022011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:27.022382 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:27.022515 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:27.023437 (+ 922us) server_negotiation.cc:299] Negotiation successful
0810 14:31:27.023683 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:31:27.025386 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.000706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41019 (local address 127.6.46.194:48481)
0810 14:31:27.001568 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:27.001620 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:27.003495 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:27.004901 (+ 1406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:27.004916 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:27.005749 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:27.006686 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.006717 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.008429 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.008435 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.010746 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.010763 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.012082 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.012089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.012378 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.013149 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:27.013177 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:27.016485 (+ 3308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:27.020513 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:27.020527 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:27.020542 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:27.021165 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:27.021609 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:27.021613 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:27.021616 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:27.021869 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:27.022589 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:27.022597 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:27.023065 (+ 468us) client_negotiation.cc:769] Sending connection context
0810 14:31:27.023318 (+ 253us) client_negotiation.cc:240] Negotiation successful
0810 14:31:27.023861 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":378,"threads_started":1}
I20260810 14:31:27.044860 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.031625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:34771 (local address 127.6.46.254:41019)
0810 14:31:27.031801 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:27.031806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:27.032527 (+ 721us) server_negotiation.cc:407] Connection header received
0810 14:31:27.032687 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:27.032692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:27.032760 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:27.032838 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:27.033711 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.034572 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.036392 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.036787 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.036890 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.039155 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:27.039160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:27.039165 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:27.039204 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:27.042287 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:27.042914 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:27.042919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:27.042924 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:27.043059 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:27.043392 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:27.043397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:27.043401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:27.043755 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:27.043882 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:27.044273 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 14:31:27.044497 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:31:27.045039 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.031749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41019 (local address 127.6.46.194:34771)
0810 14:31:27.032426 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:27.032446 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:27.032570 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:27.032878 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:27.032883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:27.033309 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:27.033583 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.033591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.034739 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.034742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.036247 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.036261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.036976 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.036983 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.037144 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.037813 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:27.037837 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:27.038922 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:27.042401 (+ 3479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:27.042406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:27.042411 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:27.042799 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:27.043165 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:27.043170 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:27.043174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:27.043293 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:27.043864 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:27.043868 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:27.044047 (+ 179us) client_negotiation.cc:769] Sending connection context
0810 14:31:27.044166 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 14:31:27.044487 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":405,"threads_started":1}
I20260810 14:31:27.045475 8584 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:34705
I20260810 14:31:27.045495 8835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:34705 every 8 connection(s)
I20260810 14:31:27.046880 8584 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:31:27.049062 8837 heartbeater.cc:347] Connected to a master server at 127.6.46.254:41019
I20260810 14:31:27.049279 8837 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:27.050052 8837 heartbeater.cc:511] Master 127.6.46.254:41019 requested a full tablet report, sending...
I20260810 14:31:27.051287 8227 ts_manager.cc:194] Registered new tserver with Master: 99e80f721f8a470d8e20ad5855510c7d (127.6.46.194:34705)
I20260810 14:31:27.052057 8227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:48481
I20260810 14:31:27.056798 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 8584
I20260810 14:31:27.056967 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:31:27.057349 6331 external_mini_cluster.cc:1428] Setting key 2f482d4d213882ba8302d1d871b3cfcd
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:31:27.097409 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:41019
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:46277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:27.414765 8848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:27.415012 8848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:27.415130 8848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:27.421988 8848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:27.422109 8848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:27.422250 8848 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:31:27.422343 8848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:31:27.431623 8848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:46277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.46.254:41019
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:27.432790 8848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:27.434106 8848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:27.446583 8860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:27.446552 8867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:27.446516 8862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:27.447320 8848 server_base.cc:1034] running on GCE node
I20260810 14:31:27.447932 8848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:27.448850 8848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:27.450091 8848 hybrid_clock.cc:648] HybridClock initialized: now 1786372287449987 us; error 47 us; skew 500 ppm
I20260810 14:31:27.450387 8848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:27 dist-test-slave-q9h9 krb5kdc[8145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372287, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:27.456148 8848 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:31:27.457476 8848 webserver.cc:460] Webserver started at http://127.6.46.195:41339/ using document root <none> and password file <none>
I20260810 14:31:27.458161 8848 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:27.458258 8848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:27.458570 8848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:27.461309 8848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "4ae7b90312134fd489ee8798eaaf0f84"
format_stamp: "Formatted at 2026-08-10 14:31:27 on dist-test-slave-q9h9"
server_key: "224bbb31f3cd30a5e286351fe4b4c3af"
server_key_iv: "ce56ee2225f173a00eefb07038128834"
server_key_version: "encryptionkey@0"
I20260810 14:31:27.462100 8848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "4ae7b90312134fd489ee8798eaaf0f84"
format_stamp: "Formatted at 2026-08-10 14:31:27 on dist-test-slave-q9h9"
server_key: "224bbb31f3cd30a5e286351fe4b4c3af"
server_key_iv: "ce56ee2225f173a00eefb07038128834"
server_key_version: "encryptionkey@0"
I20260810 14:31:27.467127 8848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 14:31:27.470434 8880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:27.471659 8848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:31:27.471805 8848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "4ae7b90312134fd489ee8798eaaf0f84"
format_stamp: "Formatted at 2026-08-10 14:31:27 on dist-test-slave-q9h9"
server_key: "224bbb31f3cd30a5e286351fe4b4c3af"
server_key_iv: "ce56ee2225f173a00eefb07038128834"
server_key_version: "encryptionkey@0"
I20260810 14:31:27.471992 8848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:27.486788 8848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:27.490252 8848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:27.490476 8848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:27.492889 8848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:27.492985 8848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:27.493129 8848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:27.493233 8848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:31:27 dist-test-slave-q9h9 krb5kdc[8145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372287, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:27.514315 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.493432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:50191 (local address 127.6.46.254:41019)
0810 14:31:27.493658 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:27.493665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:27.494966 (+ 1301us) server_negotiation.cc:407] Connection header received
0810 14:31:27.497046 (+ 2080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:27.497053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:27.497142 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:27.497253 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:27.499186 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.500297 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.502601 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.503015 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.503125 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.507479 (+ 4354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:27.507486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:27.507491 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:27.507531 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:27.510896 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:27.511757 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:27.511761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:27.511763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:27.511901 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:27.512497 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:27.512502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:27.512505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:27.512787 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:27.512906 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:27.513843 (+ 937us) server_negotiation.cc:299] Negotiation successful
0810 14:31:27.514006 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:31:27.516048 8891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.493885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41019 (local address 127.6.46.195:50191)
0810 14:31:27.494775 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:27.494831 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:27.496757 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:27.497486 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:27.497494 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:27.498202 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:27.498993 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.499019 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.500460 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.500466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.502470 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.502485 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.503258 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.503267 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.503444 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.504182 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:27.504203 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:27.507188 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:27.511027 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:27.511036 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:27.511050 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:27.511596 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:27.512016 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:27.512022 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:27.512025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:27.512354 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:27.512911 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:27.512920 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:27.513391 (+ 471us) client_negotiation.cc:769] Sending connection context
0810 14:31:27.513757 (+ 366us) client_negotiation.cc:240] Negotiation successful
0810 14:31:27.514331 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":395,"threads_started":1}
I20260810 14:31:27.535578 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.521794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:36547 (local address 127.6.46.254:41019)
0810 14:31:27.521955 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:27.521960 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:27.522545 (+ 585us) server_negotiation.cc:407] Connection header received
0810 14:31:27.522707 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:27.522710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:27.522767 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:27.522840 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:27.523998 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.524707 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.526726 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.527293 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.527412 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.529508 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:27.529513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:27.529516 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:27.529544 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:27.532538 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:27.533308 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:27.533314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:27.533318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:27.533478 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:27.533916 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:27.533921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:27.533925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:27.534323 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:27.534464 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:27.535026 (+ 562us) server_negotiation.cc:299] Negotiation successful
0810 14:31:27.535226 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:31:27.535790 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.521783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41019 (local address 127.6.46.195:36547)
0810 14:31:27.522440 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:27.522460 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:27.522611 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:27.522880 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:27.522885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:27.523466 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:27.523857 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.523870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.524812 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.524817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.526564 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.526591 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.527512 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.527520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.527719 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.528315 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:27.528337 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:27.529350 (+ 1013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:27.532671 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:27.532680 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:27.532684 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:27.533189 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:27.533624 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:27.533629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:27.533633 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:27.533809 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:27.534533 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:27.534538 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:27.534767 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 14:31:27.534912 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 14:31:27.535299 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":352,"threads_started":1}
I20260810 14:31:27.539871 8848 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:43341
I20260810 14:31:27.539891 9070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:43341 every 8 connection(s)
I20260810 14:31:27.541143 8848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:31:27.542918 9072 heartbeater.cc:347] Connected to a master server at 127.6.46.254:41019
I20260810 14:31:27.543134 9072 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:27.543737 9072 heartbeater.cc:511] Master 127.6.46.254:41019 requested a full tablet report, sending...
I20260810 14:31:27.544835 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 8848
I20260810 14:31:27.545006 8227 ts_manager.cc:194] Registered new tserver with Master: 4ae7b90312134fd489ee8798eaaf0f84 (127.6.46.195:43341)
I20260810 14:31:27.545049 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:31:27.545569 6331 external_mini_cluster.cc:1428] Setting key 0861911bd9e71a8fc8ac1f35ce9ee985
I20260810 14:31:27.546139 8227 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:50191
I20260810 14:31:27.548624 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:31:27 dist-test-slave-q9h9 krb5kdc[8145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372287, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260810 14:31:27.578243 8577 heartbeater.cc:503] Master 127.6.46.254:41019 was elected leader, sending a full tablet report...
Aug 10 14:31:27 dist-test-slave-q9h9 krb5kdc[8145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372287, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:27.597836 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.584746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58906 (local address 127.6.46.254:41019)
0810 14:31:27.584953 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:27.584960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:27.585273 (+ 313us) server_negotiation.cc:407] Connection header received
0810 14:31:27.585440 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:27.585446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:27.585513 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:27.585595 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:27.586953 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.587675 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.588962 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.589378 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.589487 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.593088 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:27.593093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:27.593096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:27.593137 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:27.595480 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:27.596038 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:27.596043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:27.596046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:27.596167 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:27.596513 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:27.596517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:27.596519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:27.596786 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:27.597021 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:27.597311 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 14:31:27.597524 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:31:27.601737 8227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:31:27.608312 8227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:31:27.631798 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.622093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51986 (local address 127.6.46.193:46501)
0810 14:31:27.623103 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:27.623112 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:27.623139 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:31:27.623232 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:27.623239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:27.623464 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:27.624430 (+ 966us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:27.624631 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.625957 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.628868 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.630341 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.630475 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.630684 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:27.631341 (+ 657us) server_negotiation.cc:299] Negotiation successful
0810 14:31:27.631514 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":1146,"threads_started":1}
I20260810 14:31:27.631907 9090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.619758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:43341 (local address 127.0.0.1:41092)
0810 14:31:27.620880 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:27.620912 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:27.621080 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:27.622918 (+ 1838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:27.622924 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:27.622954 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:27.623366 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.623378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.625732 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.625738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.627984 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.627996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.629870 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.629877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.630087 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.630159 (+ 72us) client_negotiation.cc:769] Sending connection context
0810 14:31:27.630264 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 14:31:27.630452 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"mutex_wait_us":106,"thread_start_us":778,"threads_started":1}
I20260810 14:31:27.632751 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.620521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33524 (local address 127.6.46.194:34705)
0810 14:31:27.622822 (+ 2301us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:27.622831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:27.622858 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:31:27.622954 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:27.622962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:27.623217 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:27.624245 (+ 1028us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:27.624454 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.625493 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.629917 (+ 4424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.631224 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.632029 (+ 805us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.632240 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:27.632319 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 14:31:27.632487 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2156,"thread_start_us":2027,"threads_started":1}
I20260810 14:31:27.632854 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.620104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41092 (local address 127.6.46.195:43341)
0810 14:31:27.622259 (+ 2155us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:27.622268 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:27.622303 (+ 35us) server_negotiation.cc:407] Connection header received
0810 14:31:27.622405 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:27.622412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:27.622689 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:27.623582 (+ 893us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:27.623903 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.625556 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.628148 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.629658 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.632147 (+ 2489us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.632347 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:27.632414 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 14:31:27.632554 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1986,"thread_start_us":1791,"threads_started":1}
I20260810 14:31:27.633183 9092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.619974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:34705 (local address 127.0.0.1:33524)
0810 14:31:27.621528 (+ 1554us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:27.621545 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:27.621645 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:27.623726 (+ 2081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:27.623730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:27.623780 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:27.624276 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.624290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.625648 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.625654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.629747 (+ 4093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.629762 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.631441 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.631448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.631628 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.631661 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 14:31:27.631725 (+ 64us) client_negotiation.cc:240] Negotiation successful
0810 14:31:27.631843 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1420,"mutex_wait_us":1,"spinlock_wait_cycles":3840,"thread_start_us":1257,"threads_started":1}
I20260810 14:31:27.633844 9096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.621628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:46501 (local address 127.0.0.1:51986)
0810 14:31:27.622516 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:27.622532 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:27.622631 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:27.623665 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:27.623670 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:27.623695 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:27.624227 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.624243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.626167 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.626172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.628676 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.628691 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.630818 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.630824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.630975 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.631008 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 14:31:27.631108 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 14:31:27.631228 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":1,"spinlock_wait_cycles":3456,"thread_start_us":639,"threads_started":1}
I20260810 14:31:27.636766 8957 tablet_service.cc:1467] Processing CreateTablet for tablet bf27610e62f5499cae386fa2da78725f (DEFAULT_TABLE table=test-table [id=03f25e5335c14887a0fb31bcc370bfed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:27.636853 8706 tablet_service.cc:1467] Processing CreateTablet for tablet bf27610e62f5499cae386fa2da78725f (DEFAULT_TABLE table=test-table [id=03f25e5335c14887a0fb31bcc370bfed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:27.636853 8455 tablet_service.cc:1467] Processing CreateTablet for tablet bf27610e62f5499cae386fa2da78725f (DEFAULT_TABLE table=test-table [id=03f25e5335c14887a0fb31bcc370bfed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:27.637919 8957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf27610e62f5499cae386fa2da78725f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:27.638156 8455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf27610e62f5499cae386fa2da78725f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:27.638170 8706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf27610e62f5499cae386fa2da78725f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:27.655055 9102 tablet_bootstrap.cc:492] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f: Bootstrap starting.
I20260810 14:31:27.656808 9100 tablet_bootstrap.cc:492] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84: Bootstrap starting.
I20260810 14:31:27.657972 9104 tablet_bootstrap.cc:492] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d: Bootstrap starting.
I20260810 14:31:27.660096 9100 tablet_bootstrap.cc:654] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:27.660228 9102 tablet_bootstrap.cc:654] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:27.661479 9100 log.cc:824] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:27.661788 9102 log.cc:824] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:27.662055 9104 tablet_bootstrap.cc:654] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:27.663848 9104 log.cc:824] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:27.670174 9100 tablet_bootstrap.cc:492] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84: No bootstrap required, opened a new log
I20260810 14:31:27.670310 9104 tablet_bootstrap.cc:492] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d: No bootstrap required, opened a new log
I20260810 14:31:27.670522 9100 ts_tablet_manager.cc:1397] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260810 14:31:27.670660 9102 tablet_bootstrap.cc:492] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f: No bootstrap required, opened a new log
I20260810 14:31:27.670856 9104 ts_tablet_manager.cc:1397] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.006s
I20260810 14:31:27.671041 9102 ts_tablet_manager.cc:1397] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20260810 14:31:27.679306 9100 raft_consensus.cc:348] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } }
I20260810 14:31:27.679652 9100 raft_consensus.cc:374] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:27.679780 9100 raft_consensus.cc:729] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae7b90312134fd489ee8798eaaf0f84, State: Initialized, Role: FOLLOWER
I20260810 14:31:27.680492 9100 consensus_queue.cc:260] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } }
I20260810 14:31:27.683512 9102 raft_consensus.cc:348] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } }
I20260810 14:31:27.683513 9104 raft_consensus.cc:348] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } }
I20260810 14:31:27.683987 9102 raft_consensus.cc:374] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:27.684057 9104 raft_consensus.cc:374] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:27.684191 9104 raft_consensus.cc:729] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e80f721f8a470d8e20ad5855510c7d, State: Initialized, Role: FOLLOWER
I20260810 14:31:27.684157 9102 raft_consensus.cc:729] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e17816a9ba4658ab9cc26ccc56894f, State: Initialized, Role: FOLLOWER
I20260810 14:31:27.684831 9102 consensus_queue.cc:260] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } }
I20260810 14:31:27.684855 9104 consensus_queue.cc:260] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } }
I20260810 14:31:27.688146 8837 heartbeater.cc:503] Master 127.6.46.254:41019 was elected leader, sending a full tablet report...
I20260810 14:31:27.689253 9104 ts_tablet_manager.cc:1428] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d: Time spent starting tablet: real 0.018s user 0.009s sys 0.007s
I20260810 14:31:27.691005 9072 heartbeater.cc:503] Master 127.6.46.254:41019 was elected leader, sending a full tablet report...
I20260810 14:31:27.691460 9100 ts_tablet_manager.cc:1428] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84: Time spent starting tablet: real 0.021s user 0.009s sys 0.011s
I20260810 14:31:27.694228 9102 ts_tablet_manager.cc:1428] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f: Time spent starting tablet: real 0.023s user 0.011s sys 0.012s
I20260810 14:31:27.702594 9110 raft_consensus.cc:482] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:31:27.702915 9110 raft_consensus.cc:504] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } }
I20260810 14:31:27.704586 9110 leader_election.cc:288] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99e80f721f8a470d8e20ad5855510c7d (127.6.46.194:34705), 87e17816a9ba4658ab9cc26ccc56894f (127.6.46.193:46501)
I20260810 14:31:27.710569 8891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.705007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:34705 (local address 127.6.46.195:34909)
0810 14:31:27.705196 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:27.705214 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:27.705332 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:27.705852 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:27.705856 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:27.705885 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:27.706134 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.706145 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.707216 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.707222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.708532 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.708544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.709970 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.709977 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.710187 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.710213 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:31:27.710288 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 14:31:27.710377 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 14:31:27.710796 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.704922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:34909 (local address 127.6.46.194:34705)
0810 14:31:27.705107 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:27.705112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:27.705272 (+ 160us) server_negotiation.cc:407] Connection header received
0810 14:31:27.705506 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:27.705512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:27.705587 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:27.705670 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:27.706265 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.707088 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.708685 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.709803 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.710076 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.710161 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:27.710332 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 14:31:27.710473 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:31:27.712190 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.705368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:46501 (local address 127.6.46.195:37295)
0810 14:31:27.706088 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:27.706108 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:27.706270 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:27.706604 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:27.706608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:27.706631 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:27.707117 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.707127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.708208 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.708212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.709747 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.709763 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.711577 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.711583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.711766 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.711790 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:31:27.711868 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:31:27.711961 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":393,"threads_started":1}
I20260810 14:31:27.711922 8749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf27610e62f5499cae386fa2da78725f" candidate_uuid: "4ae7b90312134fd489ee8798eaaf0f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e80f721f8a470d8e20ad5855510c7d" is_pre_election: true
I20260810 14:31:27.712271 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.705529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:37295 (local address 127.6.46.193:46501)
0810 14:31:27.705738 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:27.705744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:27.706185 (+ 441us) server_negotiation.cc:407] Connection header received
0810 14:31:27.706391 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:27.706396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:27.706498 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:27.706585 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:27.707269 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.708038 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.709958 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.711285 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.711476 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.711552 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:27.711925 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 14:31:27.712045 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:31:27.712641 8749 raft_consensus.cc:2454] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae7b90312134fd489ee8798eaaf0f84 in term 0.
I20260810 14:31:27.713416 8881 leader_election.cc:302] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae7b90312134fd489ee8798eaaf0f84, 99e80f721f8a470d8e20ad5855510c7d; no voters:
I20260810 14:31:27.713416 8498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf27610e62f5499cae386fa2da78725f" candidate_uuid: "4ae7b90312134fd489ee8798eaaf0f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e17816a9ba4658ab9cc26ccc56894f" is_pre_election: true
I20260810 14:31:27.713899 9110 raft_consensus.cc:2781] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:31:27.714089 9110 raft_consensus.cc:482] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:31:27.714213 9110 raft_consensus.cc:3037] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:27.714136 8498 raft_consensus.cc:2454] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae7b90312134fd489ee8798eaaf0f84 in term 0.
I20260810 14:31:27.716863 9110 raft_consensus.cc:504] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } }
I20260810 14:31:27.717562 9110 leader_election.cc:288] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [CANDIDATE]: Term 1 election: Requested vote from peers 99e80f721f8a470d8e20ad5855510c7d (127.6.46.194:34705), 87e17816a9ba4658ab9cc26ccc56894f (127.6.46.193:46501)
I20260810 14:31:27.717923 8749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf27610e62f5499cae386fa2da78725f" candidate_uuid: "4ae7b90312134fd489ee8798eaaf0f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e80f721f8a470d8e20ad5855510c7d"
I20260810 14:31:27.718075 8498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf27610e62f5499cae386fa2da78725f" candidate_uuid: "4ae7b90312134fd489ee8798eaaf0f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e17816a9ba4658ab9cc26ccc56894f"
I20260810 14:31:27.718120 8749 raft_consensus.cc:3037] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:27.718271 8498 raft_consensus.cc:3037] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:27.720605 8749 raft_consensus.cc:2454] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae7b90312134fd489ee8798eaaf0f84 in term 1.
I20260810 14:31:27.720620 8498 raft_consensus.cc:2454] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae7b90312134fd489ee8798eaaf0f84 in term 1.
I20260810 14:31:27.720935 8881 leader_election.cc:302] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae7b90312134fd489ee8798eaaf0f84, 99e80f721f8a470d8e20ad5855510c7d; no voters:
I20260810 14:31:27.721192 9110 raft_consensus.cc:2781] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:27.721635 9110 raft_consensus.cc:686] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [term 1 LEADER]: Becoming Leader. State: Replica: 4ae7b90312134fd489ee8798eaaf0f84, State: Running, Role: LEADER
I20260810 14:31:27.722158 9110 consensus_queue.cc:237] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } }
I20260810 14:31:27.726508 8223 catalog_manager.cc:5495] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ae7b90312134fd489ee8798eaaf0f84 (127.6.46.195). New cstate: current_term: 1 leader_uuid: "4ae7b90312134fd489ee8798eaaf0f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } health_report { overall_health: UNKNOWN } } }
I20260810 14:31:27.767987 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.762005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41098 (local address 127.6.46.195:43341)
0810 14:31:27.762230 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:27.762235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:27.762254 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:31:27.762534 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:27.762538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:27.762636 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:27.762753 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:31:27.763123 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.763860 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.765478 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.766068 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.766164 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.766794 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:31:27.767343 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:31:27.767446 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:27.767525 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 14:31:27.767595 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 14:31:27.779891 8749 raft_consensus.cc:1261] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d [term 1 FOLLOWER]: Refusing update from remote peer 4ae7b90312134fd489ee8798eaaf0f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:27.781001 9118 consensus_queue.cc:1035] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:27.782265 8498 raft_consensus.cc:1261] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f [term 1 FOLLOWER]: Refusing update from remote peer 4ae7b90312134fd489ee8798eaaf0f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:27.783123 9118 consensus_queue.cc:1035] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:31:27.792650 9073 tablet.cc:2369] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:31:27.799616 8839 tablet.cc:2369] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:31:27.800988 9128 mvcc.cc:205] Tried to move back new op lower bound from 7316980890731786240 to 7316980890513563648. Current Snapshot: MvccSnapshot[applied={T|T < 7316980890731786240 or (T in {7316980890731786240})}]
I20260810 14:31:27.801813 9127 mvcc.cc:205] Tried to move back new op lower bound from 7316980890731786240 to 7316980890513563648. Current Snapshot: MvccSnapshot[applied={T|T < 7316980890731786240}]
I20260810 14:31:27.817170 8227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:27.817472 8227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:27.821522 8227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70: Sending DeleteTablet for 3 replicas of tablet bf27610e62f5499cae386fa2da78725f
I20260810 14:31:27.822778 8957 tablet_service.cc:1514] Processing DeleteTablet for tablet bf27610e62f5499cae386fa2da78725f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:27 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:41092
I20260810 14:31:27.822978 8455 tablet_service.cc:1514] Processing DeleteTablet for tablet bf27610e62f5499cae386fa2da78725f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:27 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:51986
I20260810 14:31:27.823171 8706 tablet_service.cc:1514] Processing DeleteTablet for tablet bf27610e62f5499cae386fa2da78725f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:27 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:33524
I20260810 14:31:27.824254 8227 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:31:27.825090 8227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:31:27.832729 8454 tablet_service.cc:1467] Processing CreateTablet for tablet 654a24003ba344beacf4331d11f53563 (DEFAULT_TABLE table=test-table [id=709007660a654757b82e88cec775762a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:27.833375 8454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654a24003ba344beacf4331d11f53563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:27.833599 8705 tablet_service.cc:1467] Processing CreateTablet for tablet 654a24003ba344beacf4331d11f53563 (DEFAULT_TABLE table=test-table [id=709007660a654757b82e88cec775762a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:27.834266 8705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654a24003ba344beacf4331d11f53563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:27.838390 9142 tablet_replica.cc:330] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d: stopping tablet replica
I20260810 14:31:27.839030 9142 raft_consensus.cc:2229] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:27.839577 9142 raft_consensus.cc:2258] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:27.840744 9140 tablet_replica.cc:330] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f: stopping tablet replica
I20260810 14:31:27.841172 9140 raft_consensus.cc:2229] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:27.841745 9102 tablet_bootstrap.cc:492] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f: Bootstrap starting.
I20260810 14:31:27.842154 9104 tablet_bootstrap.cc:492] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d: Bootstrap starting.
I20260810 14:31:27.843274 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 654a24003ba344beacf4331d11f53563 (DEFAULT_TABLE table=test-table [id=709007660a654757b82e88cec775762a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:27.844182 9102 tablet_bootstrap.cc:654] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:27.844756 9104 tablet_bootstrap.cc:654] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:27.845810 9142 ts_tablet_manager.cc:1905] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:27.849771 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 654a24003ba344beacf4331d11f53563. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:27.858382 9139 tablet_replica.cc:330] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84: stopping tablet replica
I20260810 14:31:27.858958 9139 raft_consensus.cc:2229] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:31:27.859176 9102 tablet_bootstrap.cc:492] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f: No bootstrap required, opened a new log
I20260810 14:31:27.859551 9102 ts_tablet_manager.cc:1397] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20260810 14:31:27.859891 9139 raft_consensus.cc:2258] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:27.860822 9102 raft_consensus.cc:348] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:27.861177 9104 tablet_bootstrap.cc:492] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d: No bootstrap required, opened a new log
I20260810 14:31:27.861186 9102 raft_consensus.cc:374] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:27.861389 9104 ts_tablet_manager.cc:1397] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.007s
I20260810 14:31:27.861387 9102 raft_consensus.cc:729] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e17816a9ba4658ab9cc26ccc56894f, State: Initialized, Role: FOLLOWER
I20260810 14:31:27.861913 9102 consensus_queue.cc:260] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:27.862607 9102 ts_tablet_manager.cc:1428] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:31:27.862610 9104 raft_consensus.cc:348] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:27.862886 9104 raft_consensus.cc:374] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:27.862987 9104 raft_consensus.cc:729] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e80f721f8a470d8e20ad5855510c7d, State: Initialized, Role: FOLLOWER
I20260810 14:31:27.863682 9139 ts_tablet_manager.cc:1905] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:27.864452 9104 consensus_queue.cc:260] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:27.864938 9104 ts_tablet_manager.cc:1428] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:31:27.866328 9140 raft_consensus.cc:2258] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:27.866652 9100 tablet_bootstrap.cc:492] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84: Bootstrap starting.
I20260810 14:31:27.868209 9100 tablet_bootstrap.cc:654] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:27.872002 9142 ts_tablet_manager.cc:1918] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:31:27.872231 9142 log.cc:1196] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/bf27610e62f5499cae386fa2da78725f
I20260810 14:31:27.872833 9142 ts_tablet_manager.cc:1939] T bf27610e62f5499cae386fa2da78725f P 99e80f721f8a470d8e20ad5855510c7d: Deleting consensus metadata
I20260810 14:31:27.874866 8194 catalog_manager.cc:4841] TS 99e80f721f8a470d8e20ad5855510c7d (127.6.46.194:34705): tablet bf27610e62f5499cae386fa2da78725f (table test-table [id=03f25e5335c14887a0fb31bcc370bfed]) successfully deleted
I20260810 14:31:27.875792 9139 ts_tablet_manager.cc:1918] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:31:27.875955 9139 log.cc:1196] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/bf27610e62f5499cae386fa2da78725f
I20260810 14:31:27.876237 9140 ts_tablet_manager.cc:1905] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:27.876703 9139 ts_tablet_manager.cc:1939] T bf27610e62f5499cae386fa2da78725f P 4ae7b90312134fd489ee8798eaaf0f84: Deleting consensus metadata
I20260810 14:31:27.879537 8197 catalog_manager.cc:4841] TS 4ae7b90312134fd489ee8798eaaf0f84 (127.6.46.195:43341): tablet bf27610e62f5499cae386fa2da78725f (table test-table [id=03f25e5335c14887a0fb31bcc370bfed]) successfully deleted
I20260810 14:31:27.880925 9100 tablet_bootstrap.cc:492] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84: No bootstrap required, opened a new log
I20260810 14:31:27.881155 9100 ts_tablet_manager.cc:1397] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260810 14:31:27.882341 9100 raft_consensus.cc:348] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:27.882707 9100 raft_consensus.cc:374] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:27.882961 9100 raft_consensus.cc:729] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae7b90312134fd489ee8798eaaf0f84, State: Initialized, Role: FOLLOWER
I20260810 14:31:27.883244 9100 consensus_queue.cc:260] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:27.883654 9140 ts_tablet_manager.cc:1918] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:31:27.883857 9140 log.cc:1196] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/bf27610e62f5499cae386fa2da78725f
I20260810 14:31:27.883993 9100 ts_tablet_manager.cc:1428] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:31:27.884454 9140 ts_tablet_manager.cc:1939] T bf27610e62f5499cae386fa2da78725f P 87e17816a9ba4658ab9cc26ccc56894f: Deleting consensus metadata
I20260810 14:31:27.885857 8194 catalog_manager.cc:4841] TS 87e17816a9ba4658ab9cc26ccc56894f (127.6.46.193:46501): tablet bf27610e62f5499cae386fa2da78725f (table test-table [id=03f25e5335c14887a0fb31bcc370bfed]) successfully deleted
I20260810 14:31:27.971024 9114 raft_consensus.cc:482] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:31:27.971408 9114 raft_consensus.cc:504] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:27.973664 9114 leader_election.cc:288] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e17816a9ba4658ab9cc26ccc56894f (127.6.46.193:46501), 4ae7b90312134fd489ee8798eaaf0f84 (127.6.46.195:43341)
I20260810 14:31:27.984344 9149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.973778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:46501 (local address 127.6.46.194:38489)
0810 14:31:27.975429 (+ 1651us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:27.975478 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:27.975697 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:27.976102 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:27.976107 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:27.976135 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:27.977385 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.977407 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.978908 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.978912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.981979 (+ 3067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.981994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.983649 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.983653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.983774 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.983796 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:31:27.983874 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:31:27.983996 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1349,"thread_start_us":1383,"threads_started":1}
I20260810 14:31:27.984470 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.974377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:38489 (local address 127.6.46.193:46501)
0810 14:31:27.974619 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:27.974625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:27.975569 (+ 944us) server_negotiation.cc:407] Connection header received
0810 14:31:27.975880 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:27.975884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:27.975976 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:27.976075 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:27.977656 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.978700 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.982216 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.983325 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.983438 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.983489 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:27.983928 (+ 439us) server_negotiation.cc:299] Negotiation successful
0810 14:31:27.984059 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:31:27.985404 8497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654a24003ba344beacf4331d11f53563" candidate_uuid: "99e80f721f8a470d8e20ad5855510c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e17816a9ba4658ab9cc26ccc56894f" is_pre_election: true
I20260810 14:31:27.986056 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.974251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:56771 (local address 127.6.46.195:43341)
0810 14:31:27.974454 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:27.974460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:27.975785 (+ 1325us) server_negotiation.cc:407] Connection header received
0810 14:31:27.975985 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:27.975990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:27.976120 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:27.976204 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:27.977918 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.979365 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.983046 (+ 3681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.984538 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.984692 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.984783 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:27.985367 (+ 584us) server_negotiation.cc:299] Negotiation successful
0810 14:31:27.985504 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:31:27.986148 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:27.973833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:43341 (local address 127.6.46.194:56771)
0810 14:31:27.975689 (+ 1856us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:27.975716 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:27.975841 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:27.976438 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:27.976443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:27.976481 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:27.977675 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.977689 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.979522 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.979529 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.982880 (+ 3351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:27.982896 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:27.984979 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:27.984986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:27.985169 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:27.985205 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 14:31:27.985295 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 14:31:27.985442 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1707,"mutex_wait_us":1,"thread_start_us":999,"threads_started":1}
I20260810 14:31:27.986145 8497 raft_consensus.cc:2454] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e80f721f8a470d8e20ad5855510c7d in term 0.
I20260810 14:31:27.986946 8609 leader_election.cc:302] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e17816a9ba4658ab9cc26ccc56894f, 99e80f721f8a470d8e20ad5855510c7d; no voters:
I20260810 14:31:27.987336 9114 raft_consensus.cc:2781] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:31:27.987187 8983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654a24003ba344beacf4331d11f53563" candidate_uuid: "99e80f721f8a470d8e20ad5855510c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae7b90312134fd489ee8798eaaf0f84" is_pre_election: true
I20260810 14:31:27.987453 9114 raft_consensus.cc:482] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:31:27.987542 9114 raft_consensus.cc:3037] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:27.987680 8983 raft_consensus.cc:2454] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99e80f721f8a470d8e20ad5855510c7d in term 0.
I20260810 14:31:27.990379 9114 raft_consensus.cc:504] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:27.991321 9114 leader_election.cc:288] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [CANDIDATE]: Term 1 election: Requested vote from peers 87e17816a9ba4658ab9cc26ccc56894f (127.6.46.193:46501), 4ae7b90312134fd489ee8798eaaf0f84 (127.6.46.195:43341)
I20260810 14:31:27.991712 8497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654a24003ba344beacf4331d11f53563" candidate_uuid: "99e80f721f8a470d8e20ad5855510c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e17816a9ba4658ab9cc26ccc56894f"
I20260810 14:31:27.991976 8497 raft_consensus.cc:3037] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:27.994829 8497 raft_consensus.cc:2454] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e80f721f8a470d8e20ad5855510c7d in term 1.
I20260810 14:31:27.995226 8609 leader_election.cc:302] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e17816a9ba4658ab9cc26ccc56894f, 99e80f721f8a470d8e20ad5855510c7d; no voters:
I20260810 14:31:27.995324 8983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "654a24003ba344beacf4331d11f53563" candidate_uuid: "99e80f721f8a470d8e20ad5855510c7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae7b90312134fd489ee8798eaaf0f84"
I20260810 14:31:27.995468 9114 raft_consensus.cc:2781] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:27.995545 8983 raft_consensus.cc:3037] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:27.996146 9114 raft_consensus.cc:686] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [term 1 LEADER]: Becoming Leader. State: Replica: 99e80f721f8a470d8e20ad5855510c7d, State: Running, Role: LEADER
I20260810 14:31:27.996587 9114 consensus_queue.cc:237] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:27.999477 8983 raft_consensus.cc:2454] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99e80f721f8a470d8e20ad5855510c7d in term 1.
I20260810 14:31:28.001919 8223 catalog_manager.cc:5495] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 99e80f721f8a470d8e20ad5855510c7d (127.6.46.194). New cstate: current_term: 1 leader_uuid: "99e80f721f8a470d8e20ad5855510c7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } health_report { overall_health: UNKNOWN } } }
Aug 10 14:31:28 dist-test-slave-q9h9 krb5kdc[8145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372285, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
W20260810 14:31:28.075471 8578 tablet.cc:2369] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:31:28.081919 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.061749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41019 (local address 127.0.0.1:33252)
0810 14:31:28.062775 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:28.062795 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:28.063018 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:28.063441 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:28.063447 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:28.065014 (+ 1567us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:28.065552 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:28.065568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.067085 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.067091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:28.069589 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:28.069605 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.070913 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.070923 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:28.071208 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.072250 (+ 1042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:28.072277 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:28.075557 (+ 3280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:28.079348 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:28.079361 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:28.079369 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:28.079883 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:28.080273 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:28.080278 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:28.080282 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:28.080408 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:28.081037 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:28.081045 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:28.081224 (+ 179us) client_negotiation.cc:769] Sending connection context
0810 14:31:28.081343 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 14:31:28.081533 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":696,"threads_started":1}
I20260810 14:31:28.082309 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.061485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.6.46.254:41019)
0810 14:31:28.061686 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:28.061694 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:28.062900 (+ 1206us) server_negotiation.cc:407] Connection header received
0810 14:31:28.063145 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:28.063151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:28.063226 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:28.063337 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:28.065737 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.066934 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.069741 (+ 2807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.070541 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.070646 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.075783 (+ 5137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:28.075790 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:28.075793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:28.075834 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:28.079222 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:28.079985 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:28.079990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:28.079992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:28.080161 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:28.080525 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:28.080538 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:28.080542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:28.080911 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:28.081435 (+ 524us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:28.081589 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 14:31:28.081857 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 14:31:28.088358 8223 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:33252:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 14:31:28.089810 8223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:31:28.097000 8454 tablet_service.cc:1467] Processing CreateTablet for tablet 0f525108961d47d485ac283ea68114aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be06fad5af43477fbed85effd34b7cdb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:31:28.097153 8705 tablet_service.cc:1467] Processing CreateTablet for tablet 0f525108961d47d485ac283ea68114aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be06fad5af43477fbed85effd34b7cdb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:31:28.097606 8454 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f525108961d47d485ac283ea68114aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:28.097410 8956 tablet_service.cc:1467] Processing CreateTablet for tablet 0f525108961d47d485ac283ea68114aa (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=be06fad5af43477fbed85effd34b7cdb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:31:28.097642 8705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f525108961d47d485ac283ea68114aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:28.098043 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f525108961d47d485ac283ea68114aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:28.105597 9104 tablet_bootstrap.cc:492] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d: Bootstrap starting.
I20260810 14:31:28.105727 9102 tablet_bootstrap.cc:492] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f: Bootstrap starting.
I20260810 14:31:28.106432 9100 tablet_bootstrap.cc:492] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84: Bootstrap starting.
I20260810 14:31:28.107618 9102 tablet_bootstrap.cc:654] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:28.107630 9104 tablet_bootstrap.cc:654] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:28.108690 9100 tablet_bootstrap.cc:654] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:28.111404 9104 tablet_bootstrap.cc:492] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d: No bootstrap required, opened a new log
I20260810 14:31:28.111552 9104 ts_tablet_manager.cc:1397] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 14:31:28.112124 9104 raft_consensus.cc:348] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:28.112341 9104 raft_consensus.cc:374] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:28.112424 9104 raft_consensus.cc:729] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99e80f721f8a470d8e20ad5855510c7d, State: Initialized, Role: FOLLOWER
I20260810 14:31:28.112671 9104 consensus_queue.cc:260] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:28.113209 9104 ts_tablet_manager.cc:1428] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 14:31:28.113108 9114 tablet_replica.cc:439] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } }
I20260810 14:31:28.113955 9114 tablet_replica.cc:442] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:28.114967 9102 tablet_bootstrap.cc:492] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f: No bootstrap required, opened a new log
I20260810 14:31:28.115142 9100 tablet_bootstrap.cc:492] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84: No bootstrap required, opened a new log
I20260810 14:31:28.115131 9102 ts_tablet_manager.cc:1397] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 14:31:28.115291 9100 ts_tablet_manager.cc:1397] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260810 14:31:28.115849 9102 raft_consensus.cc:348] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:28.116089 9102 raft_consensus.cc:374] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:28.116176 9102 raft_consensus.cc:729] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87e17816a9ba4658ab9cc26ccc56894f, State: Initialized, Role: FOLLOWER
I20260810 14:31:28.116400 9102 consensus_queue.cc:260] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:28.117282 9100 raft_consensus.cc:348] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:28.117476 9100 raft_consensus.cc:374] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:28.117555 9100 raft_consensus.cc:729] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae7b90312134fd489ee8798eaaf0f84, State: Initialized, Role: FOLLOWER
I20260810 14:31:28.117076 9111 tablet_replica.cc:439] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } }
I20260810 14:31:28.117856 9100 consensus_queue.cc:260] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:28.118069 9102 ts_tablet_manager.cc:1428] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260810 14:31:28.118170 9111 tablet_replica.cc:442] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:28.118553 9100 ts_tablet_manager.cc:1428] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 14:31:28.119099 9110 tablet_replica.cc:439] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } }
I20260810 14:31:28.119948 9110 tablet_replica.cc:442] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:28.125243 9111 raft_consensus.cc:482] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:31:28.125438 9111 raft_consensus.cc:504] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:28.128126 9110 raft_consensus.cc:482] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:31:28.128352 9110 raft_consensus.cc:504] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:28.128310 9111 leader_election.cc:288] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99e80f721f8a470d8e20ad5855510c7d (127.6.46.194:34705), 4ae7b90312134fd489ee8798eaaf0f84 (127.6.46.195:43341)
I20260810 14:31:28.129065 9110 leader_election.cc:288] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87e17816a9ba4658ab9cc26ccc56894f (127.6.46.193:46501), 99e80f721f8a470d8e20ad5855510c7d (127.6.46.194:34705)
I20260810 14:31:28.129801 8748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f525108961d47d485ac283ea68114aa" candidate_uuid: "4ae7b90312134fd489ee8798eaaf0f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e80f721f8a470d8e20ad5855510c7d" is_pre_election: true
I20260810 14:31:28.130152 8748 raft_consensus.cc:2454] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae7b90312134fd489ee8798eaaf0f84 in term 0.
I20260810 14:31:28.130501 8497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f525108961d47d485ac283ea68114aa" candidate_uuid: "4ae7b90312134fd489ee8798eaaf0f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e17816a9ba4658ab9cc26ccc56894f" is_pre_election: true
I20260810 14:31:28.130735 8497 raft_consensus.cc:2454] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ae7b90312134fd489ee8798eaaf0f84 in term 0.
I20260810 14:31:28.131371 8881 leader_election.cc:302] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae7b90312134fd489ee8798eaaf0f84, 99e80f721f8a470d8e20ad5855510c7d; no voters:
I20260810 14:31:28.132043 9110 raft_consensus.cc:2781] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:31:28.132158 9110 raft_consensus.cc:482] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:31:28.132264 9110 raft_consensus.cc:3037] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:28.136430 9110 raft_consensus.cc:504] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:28.137236 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.127986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:46399 (local address 127.6.46.194:34705)
0810 14:31:28.128164 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:28.128171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:28.128933 (+ 762us) server_negotiation.cc:407] Connection header received
0810 14:31:28.129213 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:28.129219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:28.129294 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:28.129354 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:28.130557 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.131726 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.134824 (+ 3098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.136029 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.136145 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.136229 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:28.136832 (+ 603us) server_negotiation.cc:299] Negotiation successful
0810 14:31:28.136960 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:31:28.137452 8497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f525108961d47d485ac283ea68114aa" candidate_uuid: "4ae7b90312134fd489ee8798eaaf0f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87e17816a9ba4658ab9cc26ccc56894f"
I20260810 14:31:28.137707 8497 raft_consensus.cc:3037] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:28.138175 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.127473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:34705 (local address 127.6.46.193:46399)
0810 14:31:28.128819 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:28.128841 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:28.129037 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:28.129774 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:28.129780 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:28.129807 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:28.130368 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:28.130383 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.131897 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.131902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:28.134617 (+ 2715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:28.134632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.136451 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.136457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:28.136628 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.136676 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 14:31:28.137567 (+ 891us) client_negotiation.cc:240] Negotiation successful
0810 14:31:28.137820 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"thread_start_us":2105,"threads_started":1}
I20260810 14:31:28.138906 8748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f525108961d47d485ac283ea68114aa" candidate_uuid: "87e17816a9ba4658ab9cc26ccc56894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e80f721f8a470d8e20ad5855510c7d" is_pre_election: true
I20260810 14:31:28.139140 8748 raft_consensus.cc:2454] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87e17816a9ba4658ab9cc26ccc56894f in term 0.
I20260810 14:31:28.139925 8377 leader_election.cc:302] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87e17816a9ba4658ab9cc26ccc56894f, 99e80f721f8a470d8e20ad5855510c7d; no voters:
I20260810 14:31:28.140369 8748 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f525108961d47d485ac283ea68114aa" candidate_uuid: "4ae7b90312134fd489ee8798eaaf0f84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99e80f721f8a470d8e20ad5855510c7d"
I20260810 14:31:28.140563 8748 raft_consensus.cc:3037] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:28.140902 9110 leader_election.cc:288] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [CANDIDATE]: Term 1 election: Requested vote from peers 87e17816a9ba4658ab9cc26ccc56894f (127.6.46.193:46501), 99e80f721f8a470d8e20ad5855510c7d (127.6.46.194:34705)
I20260810 14:31:28.141669 8497 raft_consensus.cc:2454] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae7b90312134fd489ee8798eaaf0f84 in term 1.
I20260810 14:31:28.141841 9111 raft_consensus.cc:2741] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260810 14:31:28.143254 8748 raft_consensus.cc:2454] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ae7b90312134fd489ee8798eaaf0f84 in term 1.
I20260810 14:31:28.143277 8881 leader_election.cc:302] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ae7b90312134fd489ee8798eaaf0f84, 87e17816a9ba4658ab9cc26ccc56894f; no voters:
I20260810 14:31:28.143630 9110 raft_consensus.cc:2781] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:28.144240 9110 raft_consensus.cc:686] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [term 1 LEADER]: Becoming Leader. State: Replica: 4ae7b90312134fd489ee8798eaaf0f84, State: Running, Role: LEADER
I20260810 14:31:28.144557 9110 consensus_queue.cc:237] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } }
I20260810 14:31:28.146518 9118 tablet_replica.cc:439] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84: TxnStatusTablet state changed. Reason: New leader 4ae7b90312134fd489ee8798eaaf0f84. Latest consensus state: current_term: 1 leader_uuid: "4ae7b90312134fd489ee8798eaaf0f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } }
I20260810 14:31:28.147069 9118 tablet_replica.cc:442] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:31:28.148257 9174 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 14:31:28.148677 9174 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 14:31:28.148758 8223 catalog_manager.cc:5495] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ae7b90312134fd489ee8798eaaf0f84 (127.6.46.195). New cstate: current_term: 1 leader_uuid: "4ae7b90312134fd489ee8798eaaf0f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } health_report { overall_health: HEALTHY } } }
I20260810 14:31:28.149917 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.129399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:50143 (local address 127.6.46.195:43341)
0810 14:31:28.129569 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:28.129574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:28.142458 (+ 12884us) server_negotiation.cc:407] Connection header received
0810 14:31:28.142593 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:28.142598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:28.142746 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:28.142841 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:28.143333 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.144321 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.146563 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.148628 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.148824 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.148939 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:28.149493 (+ 554us) server_negotiation.cc:299] Negotiation successful
0810 14:31:28.149636 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:31:28.150667 9165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.128958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:43341 (local address 127.6.46.193:50143)
0810 14:31:28.142358 (+ 13400us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:28.142376 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:28.142487 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:28.142872 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:28.142877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:28.142904 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:28.143149 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:28.143162 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.144522 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.144527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:28.146319 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:28.146337 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.149087 (+ 2750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.149093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:28.149295 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.149330 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 14:31:28.150267 (+ 937us) client_negotiation.cc:240] Negotiation successful
0810 14:31:28.150412 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13300,"thread_start_us":10734,"threads_started":1}
I20260810 14:31:28.151384 8983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f525108961d47d485ac283ea68114aa" candidate_uuid: "87e17816a9ba4658ab9cc26ccc56894f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ae7b90312134fd489ee8798eaaf0f84" is_pre_election: true
I20260810 14:31:28.171957 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.166734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:43341 (local address 127.0.0.1:43870)
0810 14:31:28.166968 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:28.166988 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:28.167107 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:28.167441 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:28.167444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:28.167453 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:31:28.167604 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:28.167611 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.168954 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.168958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:28.169939 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:28.169950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.170947 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.170952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:28.171068 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.171121 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:31:28.171570 (+ 449us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:31:28.171580 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 14:31:28.171639 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 14:31:28.171748 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 14:31:28.172017 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.166966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43870 (local address 127.6.46.195:43341)
0810 14:31:28.167151 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:28.167157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:28.167178 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:31:28.167242 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:28.167247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:28.167327 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:28.167428 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:31:28.167744 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.168855 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.170103 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.170811 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.170936 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.171303 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:31:28.171478 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:31:28.171579 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:28.171678 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 14:31:28.171755 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:31:28.176203 8748 raft_consensus.cc:1261] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d [term 1 FOLLOWER]: Refusing update from remote peer 4ae7b90312134fd489ee8798eaaf0f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:28.176314 8497 raft_consensus.cc:1261] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f [term 1 FOLLOWER]: Refusing update from remote peer 4ae7b90312134fd489ee8798eaaf0f84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:28.176949 9118 consensus_queue.cc:1035] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:28.177233 9110 consensus_queue.cc:1035] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:28.187397 9114 tablet_replica.cc:439] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d: TxnStatusTablet state changed. Reason: New leader 4ae7b90312134fd489ee8798eaaf0f84. Latest consensus state: current_term: 1 leader_uuid: "4ae7b90312134fd489ee8798eaaf0f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } }
I20260810 14:31:28.187726 9114 tablet_replica.cc:442] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:28.187537 9111 tablet_replica.cc:439] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f: TxnStatusTablet state changed. Reason: New leader 4ae7b90312134fd489ee8798eaaf0f84. Latest consensus state: current_term: 1 leader_uuid: "4ae7b90312134fd489ee8798eaaf0f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } }
I20260810 14:31:28.187855 9111 tablet_replica.cc:442] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:28.189968 9118 tablet_replica.cc:439] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ae7b90312134fd489ee8798eaaf0f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } }
I20260810 14:31:28.190268 9118 tablet_replica.cc:442] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:31:28.190529 9126 mvcc.cc:205] Tried to move back new op lower bound from 7316980892363890688 to 7316980892243021824. Current Snapshot: MvccSnapshot[applied={T|T < 7316980892363890688}]
I20260810 14:31:28.192260 9114 tablet_replica.cc:439] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ae7b90312134fd489ee8798eaaf0f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } }
I20260810 14:31:28.192515 9114 tablet_replica.cc:442] T 0f525108961d47d485ac283ea68114aa P 99e80f721f8a470d8e20ad5855510c7d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:28.195554 9118 tablet_replica.cc:439] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4ae7b90312134fd489ee8798eaaf0f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } }
I20260810 14:31:28.195834 9118 tablet_replica.cc:442] T 0f525108961d47d485ac283ea68114aa P 4ae7b90312134fd489ee8798eaaf0f84: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:31:28.196266 9111 tablet_replica.cc:439] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4ae7b90312134fd489ee8798eaaf0f84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 } } peers { permanent_uuid: "99e80f721f8a470d8e20ad5855510c7d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 34705 } } peers { permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 } } }
I20260810 14:31:28.196499 9111 tablet_replica.cc:442] T 0f525108961d47d485ac283ea68114aa P 87e17816a9ba4658ab9cc26ccc56894f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:28.207440 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.201706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48592 (local address 127.6.46.194:34705)
0810 14:31:28.201848 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:28.201852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:28.201866 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:31:28.201942 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:28.201945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:28.202064 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:28.202193 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:31:28.202688 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.203531 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.205067 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.205638 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.205756 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.206363 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:31:28.206891 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:31:28.207016 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:28.207109 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 14:31:28.207189 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 14:31:28.225875 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.218855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:52959 (local address 127.6.46.195:43341)
0810 14:31:28.219010 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:28.219017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:28.219913 (+ 896us) server_negotiation.cc:407] Connection header received
0810 14:31:28.220096 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:28.220099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:28.220169 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:28.220241 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:31:28.221147 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.221908 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.224085 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.224572 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.224685 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.225150 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:31:28.225269 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:31:28.225344 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:28.225485 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 14:31:28.225576 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 14:31:28.225950 9185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.218773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:43341 (local address 127.6.46.194:52959)
0810 14:31:28.219810 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:28.219826 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:28.219962 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:28.220269 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:28.220273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:28.220284 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:31:28.220964 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:28.220975 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.222094 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.222098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:28.223946 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:28.223959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.224749 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.224755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:28.224876 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.224938 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:31:28.225365 (+ 427us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:31:28.225377 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 14:31:28.225445 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 14:31:28.225589 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":690,"threads_started":1}
I20260810 14:31:28.237394 8983 raft_consensus.cc:1261] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84 [term 1 FOLLOWER]: Refusing update from remote peer 99e80f721f8a470d8e20ad5855510c7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:28.238425 8497 raft_consensus.cc:1261] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f [term 1 FOLLOWER]: Refusing update from remote peer 99e80f721f8a470d8e20ad5855510c7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:28.238198 9150 consensus_queue.cc:1035] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ae7b90312134fd489ee8798eaaf0f84" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:28.238883 9114 consensus_queue.cc:1035] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "87e17816a9ba4658ab9cc26ccc56894f" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 46501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:28.280634 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.270448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:60815 (local address 127.6.46.194:34705)
0810 14:31:28.270648 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:28.270653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:28.272618 (+ 1965us) server_negotiation.cc:407] Connection header received
0810 14:31:28.272784 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:28.272788 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:28.272880 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:28.272983 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:31:28.273859 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.275324 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.277676 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.278963 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.279105 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.279613 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:31:28.279752 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:31:28.279841 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:28.280135 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 14:31:28.280238 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:31:28.281374 9192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.271048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:34705 (local address 127.6.46.195:60815)
0810 14:31:28.272513 (+ 1465us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:28.272529 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:28.272651 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:28.272998 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:28.273001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:28.273010 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:31:28.273691 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:28.273703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.275451 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.275456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:28.277532 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:28.277546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.279181 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.279189 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:28.279363 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.279440 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:31:28.279869 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:31:28.279935 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 14:31:28.280100 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 14:31:28.280252 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1257,"thread_start_us":712,"threads_started":1}
I20260810 14:31:28.328266 8227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:28.328558 8227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:28.332427 8227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dfc4b30d653e47ebaf0d3ae1f8ce8c70: Sending DeleteTablet for 3 replicas of tablet 654a24003ba344beacf4331d11f53563
I20260810 14:31:28.333495 8454 tablet_service.cc:1514] Processing DeleteTablet for tablet 654a24003ba344beacf4331d11f53563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:28 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:51986
I20260810 14:31:28.333956 9140 tablet_replica.cc:330] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f: stopping tablet replica
I20260810 14:31:28.334326 9140 raft_consensus.cc:2229] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:28.334573 9140 raft_consensus.cc:2258] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:28.335201 8956 tablet_service.cc:1514] Processing DeleteTablet for tablet 654a24003ba344beacf4331d11f53563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:28 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:41092
I20260810 14:31:28.335498 9139 tablet_replica.cc:330] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84: stopping tablet replica
I20260810 14:31:28.335765 9139 raft_consensus.cc:2229] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:28.335987 9139 raft_consensus.cc:2258] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:28.336174 9140 ts_tablet_manager.cc:1905] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:28.339331 9139 ts_tablet_manager.cc:1905] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:28.343544 8705 tablet_service.cc:1514] Processing DeleteTablet for tablet 654a24003ba344beacf4331d11f53563 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:28 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:33524
I20260810 14:31:28.343894 9142 tablet_replica.cc:330] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d: stopping tablet replica
I20260810 14:31:28.344132 9142 raft_consensus.cc:2229] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:31:28.344720 9142 raft_consensus.cc:2258] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:28.346240 9142 ts_tablet_manager.cc:1905] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:28.352582 9139 ts_tablet_manager.cc:1918] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:31:28.353917 9139 log.cc:1196] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/654a24003ba344beacf4331d11f53563
I20260810 14:31:28.354701 9139 ts_tablet_manager.cc:1939] T 654a24003ba344beacf4331d11f53563 P 4ae7b90312134fd489ee8798eaaf0f84: Deleting consensus metadata
I20260810 14:31:28.355370 9140 ts_tablet_manager.cc:1918] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:31:28.355544 9140 log.cc:1196] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/654a24003ba344beacf4331d11f53563
I20260810 14:31:28.355996 9140 ts_tablet_manager.cc:1939] T 654a24003ba344beacf4331d11f53563 P 87e17816a9ba4658ab9cc26ccc56894f: Deleting consensus metadata
I20260810 14:31:28.357224 8194 catalog_manager.cc:4841] TS 87e17816a9ba4658ab9cc26ccc56894f (127.6.46.193:46501): tablet 654a24003ba344beacf4331d11f53563 (table test-table [id=709007660a654757b82e88cec775762a]) successfully deleted
I20260810 14:31:28.357743 8197 catalog_manager.cc:4841] TS 4ae7b90312134fd489ee8798eaaf0f84 (127.6.46.195:43341): tablet 654a24003ba344beacf4331d11f53563 (table test-table [id=709007660a654757b82e88cec775762a]) successfully deleted
I20260810 14:31:28.353978 9142 ts_tablet_manager.cc:1918] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:31:28.358116 9142 log.cc:1196] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/654a24003ba344beacf4331d11f53563
I20260810 14:31:28.358867 9142 ts_tablet_manager.cc:1939] T 654a24003ba344beacf4331d11f53563 P 99e80f721f8a470d8e20ad5855510c7d: Deleting consensus metadata
I20260810 14:31:28.360023 8194 catalog_manager.cc:4841] TS 99e80f721f8a470d8e20ad5855510c7d (127.6.46.194:34705): tablet 654a24003ba344beacf4331d11f53563 (table test-table [id=709007660a654757b82e88cec775762a]) successfully deleted
Aug 10 14:31:28 dist-test-slave-q9h9 krb5kdc[8145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372287, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:31:28.380992 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.359975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.6.46.193:46501)
0810 14:31:28.360165 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:28.360171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:28.361419 (+ 1248us) server_negotiation.cc:407] Connection header received
0810 14:31:28.361613 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:28.361619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:28.361704 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:28.361811 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:28.364016 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.365255 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.367181 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.367842 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.367969 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.373203 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:28.373226 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:28.373246 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:28.373293 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:28.377508 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:28.378437 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:28.378453 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:28.378456 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:28.378648 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:28.379081 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:28.379087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:28.379090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:28.379562 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:28.379882 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:28.380365 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 14:31:28.380615 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 14:31:28.382184 8570 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46372
I20260810 14:31:28.406705 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.386555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33256 (local address 127.6.46.254:41019)
0810 14:31:28.386770 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:28.386776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:28.390516 (+ 3740us) server_negotiation.cc:407] Connection header received
0810 14:31:28.390708 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:28.390713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:28.390801 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:28.390900 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:28.392865 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.393985 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.396175 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.396881 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.397002 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.399951 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:28.399957 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:28.399961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:28.400001 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:28.403442 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:28.404304 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:28.404310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:28.404313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:28.404477 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:28.404962 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:28.404966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:28.404969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:28.405350 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:28.405616 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:28.406117 (+ 501us) server_negotiation.cc:299] Negotiation successful
0810 14:31:28.406354 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20260810 14:31:28.407342 8227 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33256
I20260810 14:31:28.410115 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8348
I20260810 14:31:28.427964 9099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.412072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.6.46.193:46501)
0810 14:31:28.412258 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:28.412264 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:28.412701 (+ 437us) server_negotiation.cc:407] Connection header received
0810 14:31:28.412969 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:28.412975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:28.413089 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:28.413202 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:28.414657 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.415798 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.418053 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.418643 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.418768 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.421619 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:28.421626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:28.421631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:28.421672 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:28.425034 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:28.425816 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:28.425821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:28.425825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:28.425969 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:28.426367 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:28.426372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:28.426375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:28.426705 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:28.426962 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:28.427297 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 14:31:28.427601 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 14:31:28.428529 8570 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46386
W20260810 14:31:28.429122 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8348: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:31:28.429292 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 8348
I20260810 14:31:28.461397 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8584
Aug 10 14:31:28 dist-test-slave-q9h9 krb5kdc[8145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372287, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:31:28.481506 9094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.462313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48606 (local address 127.6.46.194:34705)
0810 14:31:28.462520 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:28.462530 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:28.462897 (+ 367us) server_negotiation.cc:407] Connection header received
0810 14:31:28.463106 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:28.463112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:28.463210 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:28.463328 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:28.464918 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.466269 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.468109 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.468778 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.468919 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.474044 (+ 5125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:28.474060 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:28.474081 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:28.474123 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:28.478142 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:28.478940 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:28.478949 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:28.478953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:28.479153 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:28.479629 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:28.479635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:28.479638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:28.480079 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:28.480443 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:28.480735 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 14:31:28.480951 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 14:31:28.482295 8827 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48606
W20260810 14:31:28.482970 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8584: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:31:28.483161 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 8584
I20260810 14:31:28.514837 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8848
Aug 10 14:31:28 dist-test-slave-q9h9 krb5kdc[8145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372287, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:31:28.530305 9093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:28.515548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.6.46.195:43341)
0810 14:31:28.515742 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:28.515747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:28.515832 (+ 85us) server_negotiation.cc:407] Connection header received
0810 14:31:28.516017 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:28.516023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:28.516122 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:28.516210 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:28.517427 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.518315 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.519563 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:28.520142 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:28.520250 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:28.524379 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:28.524395 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:28.524411 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:28.524460 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:28.527599 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:28.528236 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:28.528244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:28.528246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:28.528404 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:28.528753 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:28.528757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:28.528759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:28.529126 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:28.529448 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:28.529727 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 14:31:28.529957 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260810 14:31:28.530949 9064 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43882
W20260810 14:31:28.531567 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8848: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:31:28.531701 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 8848
I20260810 14:31:28.560781 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8166
I20260810 14:31:28.561910 8296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:28.675047 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 8166
2026-08-10T14:31:28Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4474 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:31:28 dist-test-slave-q9h9 krb5kdc[9213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:31:28 dist-test-slave-q9h9 krb5kdc[9213](info): set up 2 sockets
Aug 10 14:31:28 dist-test-slave-q9h9 krb5kdc[9213](info): commencing operation
krb5kdc: starting...
W20260810 14:31:29.940238 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260810 14:31:30.099437 6331 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.159s user 0.000s sys 0.007s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260810 14:31:30.244359 6331 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.145s user 0.000s sys 0.006s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260810 14:31:30.369343 6331 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.125s user 0.000s sys 0.006s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/test-user.keytab.
W20260810 14:31:31.034402 6331 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.665s user 0.000s sys 0.007s
Aug 10 14:31:31 dist-test-slave-q9h9 krb5kdc[9213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:31:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:31:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
W20260810 14:31:31.417392 6331 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.46.254: real 0.333s user 0.001s sys 0.010s
I20260810 14:31:31.418533 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:34309
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:34309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:31.739652 9234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:31.739930 9234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:31.740073 9234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:31.747056 9234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:31:31.747164 9234 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 14:31:31.747232 9234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:31.747315 9234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:31.747382 9234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:31:31.747466 9234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:31:31.756816 9234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:34309
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:34309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:31.758000 9234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:31.759330 9234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:31.770401 9245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:31.770447 9243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:31.770365 9248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:31.771252 9234 server_base.cc:1034] running on GCE node
I20260810 14:31:31.771953 9234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:31.773244 9234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:31.774464 9234 hybrid_clock.cc:648] HybridClock initialized: now 1786372291774403 us; error 32 us; skew 500 ppm
I20260810 14:31:31.774784 9234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:31 dist-test-slave-q9h9 krb5kdc[9213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372291, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:31.777837 9234 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:31:31.779270 9234 webserver.cc:460] Webserver started at http://127.6.46.254:44089/ using document root <none> and password file <none>
I20260810 14:31:31.779887 9234 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:31.779973 9234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:31.780229 9234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:31.783058 9234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "f79ed22c59fd4f099e18bbced4042744"
format_stamp: "Formatted at 2026-08-10 14:31:31 on dist-test-slave-q9h9"
server_key: "7ce60a49cb626a1d7d71ca3e36260c38"
server_key_iv: "d1f9a8e4b862a3f63017fd757e87de33"
server_key_version: "encryptionkey@0"
I20260810 14:31:31.783756 9234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "f79ed22c59fd4f099e18bbced4042744"
format_stamp: "Formatted at 2026-08-10 14:31:31 on dist-test-slave-q9h9"
server_key: "7ce60a49cb626a1d7d71ca3e36260c38"
server_key_iv: "d1f9a8e4b862a3f63017fd757e87de33"
server_key_version: "encryptionkey@0"
I20260810 14:31:31.788563 9234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:31:31.791780 9255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:31.793026 9234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260810 14:31:31.793167 9234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "f79ed22c59fd4f099e18bbced4042744"
format_stamp: "Formatted at 2026-08-10 14:31:31 on dist-test-slave-q9h9"
server_key: "7ce60a49cb626a1d7d71ca3e36260c38"
server_key_iv: "d1f9a8e4b862a3f63017fd757e87de33"
server_key_version: "encryptionkey@0"
I20260810 14:31:31.793354 9234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:31.824802 9234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:31.828470 9234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:31.828718 9234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:31.848390 9234 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:34309
I20260810 14:31:31.848398 9365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:34309 every 8 connection(s)
I20260810 14:31:31.849642 9234 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:31:31.853211 9369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:31.856954 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 9234
I20260810 14:31:31.857182 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:31:31.857803 6331 external_mini_cluster.cc:1428] Setting key 56cc2063e1484037575be0141c0c2612
I20260810 14:31:31.863747 9369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744: Bootstrap starting.
I20260810 14:31:31.867144 9369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:31.868244 9369 log.cc:824] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:31.870800 9369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744: No bootstrap required, opened a new log
Aug 10 14:31:31 dist-test-slave-q9h9 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:31.878315 9369 raft_consensus.cc:348] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79ed22c59fd4f099e18bbced4042744" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34309 } }
I20260810 14:31:31.878665 9369 raft_consensus.cc:374] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:31.878755 9369 raft_consensus.cc:729] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f79ed22c59fd4f099e18bbced4042744, State: Initialized, Role: FOLLOWER
I20260810 14:31:31.879117 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:31.859656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.6.46.254:34309)
0810 14:31:31.860348 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:31.860359 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:31.860406 (+ 47us) server_negotiation.cc:407] Connection header received
0810 14:31:31.862310 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:31.862330 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:31.862678 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:31.862962 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:31.864762 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:31.866074 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:31.867694 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:31.868334 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:31.868523 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:31.872239 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:31.872250 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:31.872263 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:31.872307 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:31.875298 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:31.875923 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:31.875929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:31.875934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:31.876101 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:31.876466 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:31.876470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:31.876472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:31.876987 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:31.877265 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:31.877713 (+ 448us) server_negotiation.cc:299] Negotiation successful
0810 14:31:31.878115 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":310,"threads_started":1}
I20260810 14:31:31.879303 9369 consensus_queue.cc:260] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79ed22c59fd4f099e18bbced4042744" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34309 } }
I20260810 14:31:31.879540 9369 raft_consensus.cc:388] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:31:31.879689 9369 raft_consensus.cc:482] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:31:31.879842 9369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:31.882165 9369 raft_consensus.cc:504] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79ed22c59fd4f099e18bbced4042744" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34309 } }
I20260810 14:31:31.882599 9369 leader_election.cc:302] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f79ed22c59fd4f099e18bbced4042744; no voters:
I20260810 14:31:31.883116 9369 leader_election.cc:288] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:31:31.883240 9376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:31.883900 9376 raft_consensus.cc:686] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [term 1 LEADER]: Becoming Leader. State: Replica: f79ed22c59fd4f099e18bbced4042744, State: Running, Role: LEADER
I20260810 14:31:31.884403 9376 consensus_queue.cc:237] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79ed22c59fd4f099e18bbced4042744" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34309 } }
I20260810 14:31:31.884899 9369 sys_catalog.cc:564] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:31:31.887008 9378 sys_catalog.cc:455] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f79ed22c59fd4f099e18bbced4042744. Latest consensus state: current_term: 1 leader_uuid: "f79ed22c59fd4f099e18bbced4042744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79ed22c59fd4f099e18bbced4042744" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34309 } } }
I20260810 14:31:31.887773 9378 sys_catalog.cc:458] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:31.887452 9377 sys_catalog.cc:455] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f79ed22c59fd4f099e18bbced4042744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f79ed22c59fd4f099e18bbced4042744" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34309 } } }
I20260810 14:31:31.888495 9377 sys_catalog.cc:458] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:31.889374 9391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:31:31.893586 9391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:31:31.900842 9391 catalog_manager.cc:1347] Generated new cluster ID: 370da8742f924b69bbe5470e18a4a80e
I20260810 14:31:31.900933 9391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:31:31.909554 9391 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:31:31.911307 9391 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:31:31.919209 9391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744: Generated new TSK 0
I20260810 14:31:31.920105 9391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:31:31.965595 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:34309
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:32.280467 9411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:32.280715 9411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:32.280840 9411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:32.287964 9411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:32.288075 9411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:32.288179 9411 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 14:31:32.288237 9411 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:31:32.288338 9411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:31:32.297771 9411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.46.254:34309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:32.298978 9411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:32.300360 9411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:32.312639 9418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:32.312757 9416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:32.312594 9421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:32.313659 9411 server_base.cc:1034] running on GCE node
I20260810 14:31:32.314292 9411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:32.315148 9411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:32.316387 9411 hybrid_clock.cc:648] HybridClock initialized: now 1786372292316320 us; error 42 us; skew 500 ppm
I20260810 14:31:32.316701 9411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:32 dist-test-slave-q9h9 krb5kdc[9213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372292, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:32.319757 9411 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:31:32.321089 9411 webserver.cc:460] Webserver started at http://127.6.46.193:36689/ using document root <none> and password file <none>
I20260810 14:31:32.321702 9411 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:32.321837 9411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:32.322180 9411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:32.324926 9411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "bd697cce9db247adb522296cbbf1d939"
format_stamp: "Formatted at 2026-08-10 14:31:32 on dist-test-slave-q9h9"
server_key: "4afec692a0065644cf023f7ac4c7cc6b"
server_key_iv: "4eb45d74b70ebf856d371f6a5daa4d66"
server_key_version: "encryptionkey@0"
I20260810 14:31:32.325640 9411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "bd697cce9db247adb522296cbbf1d939"
format_stamp: "Formatted at 2026-08-10 14:31:32 on dist-test-slave-q9h9"
server_key: "4afec692a0065644cf023f7ac4c7cc6b"
server_key_iv: "4eb45d74b70ebf856d371f6a5daa4d66"
server_key_version: "encryptionkey@0"
I20260810 14:31:32.330436 9411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260810 14:31:32.333702 9432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:32.334929 9411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:31:32.335059 9411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "bd697cce9db247adb522296cbbf1d939"
format_stamp: "Formatted at 2026-08-10 14:31:32 on dist-test-slave-q9h9"
server_key: "4afec692a0065644cf023f7ac4c7cc6b"
server_key_iv: "4eb45d74b70ebf856d371f6a5daa4d66"
server_key_version: "encryptionkey@0"
I20260810 14:31:32.335220 9411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:32.362782 9411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:32.366360 9411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:32.366619 9411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:32.369032 9411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:32.369154 9411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:32.369262 9411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:32.369349 9411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:31:32 dist-test-slave-q9h9 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372292, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:32.390693 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:32.369740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:35993 (local address 127.6.46.254:34309)
0810 14:31:32.369896 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:32.369901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:32.371328 (+ 1427us) server_negotiation.cc:407] Connection header received
0810 14:31:32.373365 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:32.373371 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:32.373456 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:32.373543 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:32.375160 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.376271 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.378534 (+ 2263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.379093 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.379228 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:32.383739 (+ 4511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:32.383746 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:32.383752 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:32.383798 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:32.387084 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:32.388100 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:32.388105 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:32.388108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:32.388262 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:32.388863 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:32.388868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:32.388871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:32.389277 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:32.389457 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:32.390278 (+ 821us) server_negotiation.cc:299] Negotiation successful
0810 14:31:32.390447 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:31:32.392097 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:32.370391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:34309 (local address 127.6.46.193:35993)
0810 14:31:32.371166 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:32.371210 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:32.373073 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:32.373799 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:32.373807 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:32.374339 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:32.374956 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:32.374982 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.376467 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.376471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:32.378373 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:32.378396 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.379351 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.379359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:32.379641 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:32.380485 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:32.380512 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:32.383480 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:32.387274 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:32.387282 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:32.387299 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:32.387950 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:32.388421 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:32.388426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:32.388433 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:32.388739 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:32.389425 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:32.389434 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:32.389868 (+ 434us) client_negotiation.cc:769] Sending connection context
0810 14:31:32.390194 (+ 326us) client_negotiation.cc:240] Negotiation successful
0810 14:31:32.390707 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":360,"threads_started":1}
I20260810 14:31:32.411033 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:32.397403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:57017 (local address 127.6.46.254:34309)
0810 14:31:32.397568 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:32.397573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:32.398398 (+ 825us) server_negotiation.cc:407] Connection header received
0810 14:31:32.398592 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:32.398596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:32.398666 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:32.398756 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:32.400177 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.400859 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.402855 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.403253 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.403344 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:32.405529 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:32.405536 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:32.405541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:32.405574 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:32.408302 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:32.409120 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:32.409125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:32.409128 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:32.409275 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:32.409634 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:32.409639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:32.409642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:32.409881 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:32.410042 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:32.410453 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 14:31:32.410652 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:31:32.411083 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:32.397526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:34309 (local address 127.6.46.193:57017)
0810 14:31:32.398297 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:32.398316 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:32.398450 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:32.398800 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:32.398805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:32.399471 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:32.399930 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:32.399947 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.401004 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.401009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:32.402712 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:32.402729 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.403428 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.403435 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:32.403638 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:32.404196 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:32.404217 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:32.405354 (+ 1137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:32.408433 (+ 3079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:32.408438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:32.408442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:32.408987 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:32.409397 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:32.409401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:32.409405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:32.409548 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:32.409993 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:32.409998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:32.410214 (+ 216us) client_negotiation.cc:769] Sending connection context
0810 14:31:32.410361 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 14:31:32.410676 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":431,"threads_started":1}
I20260810 14:31:32.425072 9411 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:34451
I20260810 14:31:32.425110 9648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:34451 every 8 connection(s)
I20260810 14:31:32.426220 9411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:31:32.428030 9650 heartbeater.cc:347] Connected to a master server at 127.6.46.254:34309
I20260810 14:31:32.428215 9650 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:32.428742 9650 heartbeater.cc:511] Master 127.6.46.254:34309 requested a full tablet report, sending...
I20260810 14:31:32.430500 9285 ts_manager.cc:194] Registered new tserver with Master: bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451)
I20260810 14:31:32.431800 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 9411
I20260810 14:31:32.431869 9285 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:35993
I20260810 14:31:32.431998 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:31:32.432664 6331 external_mini_cluster.cc:1428] Setting key 60d4ecb88a2c7c6ee5281550eeede641
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:31:32.470803 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:34309
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:32.782159 9659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:32.782403 9659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:32.782531 9659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:32.791275 9659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:32.791404 9659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:32.791499 9659 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 14:31:32.791555 9659 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:31:32.791639 9659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:31:32.803135 9659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.46.254:34309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:32.804272 9659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:32.805474 9659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:32.817598 9664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:32.817632 9671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:32.817679 9666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:32.818810 9659 server_base.cc:1034] running on GCE node
I20260810 14:31:32.819402 9659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:32.820302 9659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:32.821518 9659 hybrid_clock.cc:648] HybridClock initialized: now 1786372292821462 us; error 31 us; skew 500 ppm
I20260810 14:31:32.821828 9659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:32 dist-test-slave-q9h9 krb5kdc[9213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372292, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:32.827996 9659 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:31:32.829418 9659 webserver.cc:460] Webserver started at http://127.6.46.194:43229/ using document root <none> and password file <none>
I20260810 14:31:32.830084 9659 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:32.830217 9659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:32.830564 9659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:32.833699 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "f3e9412ec942418d8aa01581235ffbd0"
format_stamp: "Formatted at 2026-08-10 14:31:32 on dist-test-slave-q9h9"
server_key: "7316e428716ce80658fb9bff306f4bca"
server_key_iv: "173f18efa9ec1ea6995be7cb88a2fedc"
server_key_version: "encryptionkey@0"
I20260810 14:31:32.834415 9659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "f3e9412ec942418d8aa01581235ffbd0"
format_stamp: "Formatted at 2026-08-10 14:31:32 on dist-test-slave-q9h9"
server_key: "7316e428716ce80658fb9bff306f4bca"
server_key_iv: "173f18efa9ec1ea6995be7cb88a2fedc"
server_key_version: "encryptionkey@0"
I20260810 14:31:32.839216 9659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:31:32.842523 9679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:32.843770 9659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 14:31:32.843923 9659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "f3e9412ec942418d8aa01581235ffbd0"
format_stamp: "Formatted at 2026-08-10 14:31:32 on dist-test-slave-q9h9"
server_key: "7316e428716ce80658fb9bff306f4bca"
server_key_iv: "173f18efa9ec1ea6995be7cb88a2fedc"
server_key_version: "encryptionkey@0"
I20260810 14:31:32.844097 9659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:32.861397 9659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:32.866006 9659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:32.866369 9659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:32.869071 9659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:32.869196 9659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:32.869325 9659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:32.869424 9659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:31:32 dist-test-slave-q9h9 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372292, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:32.890525 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:32.869554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:58937 (local address 127.6.46.254:34309)
0810 14:31:32.869732 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:32.869739 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:32.871051 (+ 1312us) server_negotiation.cc:407] Connection header received
0810 14:31:32.873061 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:32.873067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:32.873151 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:32.873269 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:32.875011 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.876122 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.878728 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.879374 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.879496 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:32.884160 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:32.884167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:32.884172 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:32.884212 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:32.887056 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:32.888021 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:32.888027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:32.888030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:32.888187 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:32.888751 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:32.888756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:32.888760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:32.889049 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:32.889165 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:32.890142 (+ 977us) server_negotiation.cc:299] Negotiation successful
0810 14:31:32.890313 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 14:31:32.892131 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:32.870163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:34309 (local address 127.6.46.194:58937)
0810 14:31:32.870907 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:32.870941 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:32.872728 (+ 1787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:32.873496 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:32.873507 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:32.874294 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:32.874844 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:32.874861 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.876306 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.876312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:32.878560 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:32.878579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.879560 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.879566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:32.879774 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:32.880522 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:32.880557 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:32.883881 (+ 3324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:32.887196 (+ 3315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:32.887205 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:32.887220 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:32.887882 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:32.888310 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:32.888319 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:32.888323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:32.888616 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:32.889178 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:32.889187 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:32.889667 (+ 480us) client_negotiation.cc:769] Sending connection context
0810 14:31:32.890005 (+ 338us) client_negotiation.cc:240] Negotiation successful
0810 14:31:32.890565 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":350,"threads_started":1}
I20260810 14:31:32.909984 9659 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:42141
I20260810 14:31:32.910058 9882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:42141 every 8 connection(s)
I20260810 14:31:32.911521 9659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:31:32.913645 9883 heartbeater.cc:347] Connected to a master server at 127.6.46.254:34309
I20260810 14:31:32.913844 9883 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:32.914512 9883 heartbeater.cc:511] Master 127.6.46.254:34309 requested a full tablet report, sending...
I20260810 14:31:32.915024 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:32.898665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:38193 (local address 127.6.46.254:34309)
0810 14:31:32.898848 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:32.898855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:32.901772 (+ 2917us) server_negotiation.cc:407] Connection header received
0810 14:31:32.901950 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:32.901956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:32.902057 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:32.902148 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:32.903046 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.904125 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.906212 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.906849 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.906973 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:32.909011 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:32.909017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:32.909021 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:32.909056 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:32.912217 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:32.913084 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:32.913089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:32.913092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:32.913213 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:32.913548 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:32.913552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:32.913555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:32.913912 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:32.914071 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:32.914458 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 14:31:32.914678 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 14:31:32.915112 9839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:32.899017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:34309 (local address 127.6.46.194:38193)
0810 14:31:32.901651 (+ 2634us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:32.901675 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:32.901823 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:32.902211 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:32.902216 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:32.902615 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:32.902911 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:32.902921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.904275 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.904280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:32.905996 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:32.906035 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:32.907035 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:32.907042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:32.907229 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:32.907924 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:32.907949 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:32.908819 (+ 870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:32.912377 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:32.912382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:32.912385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:32.912929 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:32.913316 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:32.913319 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:32.913321 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:32.913442 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:32.914035 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:32.914040 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:32.914228 (+ 188us) client_negotiation.cc:769] Sending connection context
0810 14:31:32.914354 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 14:31:32.914664 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2487,"thread_start_us":2364,"threads_started":1}
I20260810 14:31:32.915993 9284 ts_manager.cc:194] Registered new tserver with Master: f3e9412ec942418d8aa01581235ffbd0 (127.6.46.194:42141)
I20260810 14:31:32.916572 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 9659
I20260810 14:31:32.916778 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:31:32.917063 9284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:58937
I20260810 14:31:32.917309 6331 external_mini_cluster.cc:1428] Setting key 593cce025b46c22c72d1b1d51a4561e0
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:31:32.956481 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:34309
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:33.265057 9893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:33.265318 9893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:33.265425 9893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:33.272342 9893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:33.272436 9893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:33.272518 9893 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 14:31:33.272572 9893 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:31:33.272640 9893 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:31:33.281793 9893 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.46.254:34309
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:33.282900 9893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:33.284062 9893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:33.296233 9902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:33.296271 9905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:33.296303 9909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:33.297035 9893 server_base.cc:1034] running on GCE node
I20260810 14:31:33.297596 9893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:33.298529 9893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:33.299721 9893 hybrid_clock.cc:648] HybridClock initialized: now 1786372293299668 us; error 49 us; skew 500 ppm
I20260810 14:31:33.300000 9893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:33 dist-test-slave-q9h9 krb5kdc[9213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372293, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:33.302966 9893 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:31:33.304316 9893 webserver.cc:460] Webserver started at http://127.6.46.195:42161/ using document root <none> and password file <none>
I20260810 14:31:33.304915 9893 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:33.305032 9893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:33.305320 9893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:33.308310 9893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a"
format_stamp: "Formatted at 2026-08-10 14:31:33 on dist-test-slave-q9h9"
server_key: "1cda0bb7d4866bcf4ba13d6e923f74b6"
server_key_iv: "9bc606c79034e5db3b1fc9564077d82f"
server_key_version: "encryptionkey@0"
I20260810 14:31:33.309013 9893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a"
format_stamp: "Formatted at 2026-08-10 14:31:33 on dist-test-slave-q9h9"
server_key: "1cda0bb7d4866bcf4ba13d6e923f74b6"
server_key_iv: "9bc606c79034e5db3b1fc9564077d82f"
server_key_version: "encryptionkey@0"
I20260810 14:31:33.313787 9893 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260810 14:31:33.316998 9920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:33.318236 9893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:31:33.318382 9893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a"
format_stamp: "Formatted at 2026-08-10 14:31:33 on dist-test-slave-q9h9"
server_key: "1cda0bb7d4866bcf4ba13d6e923f74b6"
server_key_iv: "9bc606c79034e5db3b1fc9564077d82f"
server_key_version: "encryptionkey@0"
I20260810 14:31:33.318554 9893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:33.356498 9893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:33.360183 9893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:33.360419 9893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:33.362879 9893 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:33.363003 9893 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:33.363116 9893 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:33.363231 9893 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:31:33 dist-test-slave-q9h9 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372293, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:33.386063 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.363444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:44571 (local address 127.6.46.254:34309)
0810 14:31:33.363645 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:33.363652 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:33.365406 (+ 1754us) server_negotiation.cc:407] Connection header received
0810 14:31:33.367670 (+ 2264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:33.367675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:33.367768 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:33.367892 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:33.370257 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.371384 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.373545 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.374191 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.374320 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.379030 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:33.379037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:33.379041 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:33.379085 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:33.382238 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:33.383179 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:33.383185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:33.383188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:33.383349 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:33.383922 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:33.383927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:33.383930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:33.384266 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:33.384419 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:33.385392 (+ 973us) server_negotiation.cc:299] Negotiation successful
0810 14:31:33.385682 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:31:33.387228 9938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.364243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:34309 (local address 127.6.46.195:44571)
0810 14:31:33.365224 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:33.365281 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:33.367374 (+ 2093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:33.368221 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:33.368232 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:33.369027 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:33.369966 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.369997 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.371543 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.371548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.373404 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.373416 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.374610 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.374616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.374902 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.375593 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:33.375628 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:33.378733 (+ 3105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:33.382401 (+ 3668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:33.382410 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:33.382420 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:33.382997 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:33.383460 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:33.383465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:33.383473 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:33.383758 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:33.384516 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:33.384525 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:33.385005 (+ 480us) client_negotiation.cc:769] Sending connection context
0810 14:31:33.385278 (+ 273us) client_negotiation.cc:240] Negotiation successful
0810 14:31:33.385751 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":420,"threads_started":1}
I20260810 14:31:33.401472 9893 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:42153
I20260810 14:31:33.401530 10128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:42153 every 8 connection(s)
I20260810 14:31:33.403074 9893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:31:33.405259 10132 heartbeater.cc:347] Connected to a master server at 127.6.46.254:34309
I20260810 14:31:33.405488 10132 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:33.406289 10132 heartbeater.cc:511] Master 127.6.46.254:34309 requested a full tablet report, sending...
I20260810 14:31:33.407590 9284 ts_manager.cc:194] Registered new tserver with Master: 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153)
I20260810 14:31:33.407958 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.394127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:34309 (local address 127.6.46.195:49385)
0810 14:31:33.395140 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:33.395164 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:33.395408 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:33.395748 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:33.395753 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:33.396386 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:33.396851 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.396867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.398100 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.398109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.399674 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.399687 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.400426 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.400434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.400573 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.401191 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:33.401214 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:33.402087 (+ 873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:33.405455 (+ 3368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:33.405460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:33.405463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:33.405980 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:33.406381 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:33.406386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:33.406389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:33.406539 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:33.406990 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:33.406996 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:33.407145 (+ 149us) client_negotiation.cc:769] Sending connection context
0810 14:31:33.407257 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 14:31:33.407533 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":519,"threads_started":1}
I20260810 14:31:33.407994 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.394049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:49385 (local address 127.6.46.254:34309)
0810 14:31:33.394226 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:33.394231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:33.395251 (+ 1020us) server_negotiation.cc:407] Connection header received
0810 14:31:33.395540 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:33.395545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:33.395608 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:33.395710 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:33.396994 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.397885 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.399815 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.400237 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.400344 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.402247 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:33.402254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:33.402259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:33.402296 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:33.405358 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:33.406126 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:33.406130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:33.406132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:33.406271 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:33.406645 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:33.406648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:33.406650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:33.406879 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:33.406995 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:33.407432 (+ 437us) server_negotiation.cc:299] Negotiation successful
0810 14:31:33.407628 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:31:33.408546 9284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:44571
I20260810 14:31:33.411389 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 9893
I20260810 14:31:33.411572 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:31:33.411948 6331 external_mini_cluster.cc:1428] Setting key 36f0219dfeac41e5618b1744b8155e9c
I20260810 14:31:33.414629 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:31:33.434516 9650 heartbeater.cc:503] Master 127.6.46.254:34309 was elected leader, sending a full tablet report...
Aug 10 14:31:33 dist-test-slave-q9h9 krb5kdc[9213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:31:33 dist-test-slave-q9h9 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:33.460769 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.448483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.6.46.254:34309)
0810 14:31:33.448667 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:33.448673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:33.448947 (+ 274us) server_negotiation.cc:407] Connection header received
0810 14:31:33.449079 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:33.449082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:33.449143 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:33.449220 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:33.450552 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.451274 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.452398 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.452818 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.452904 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.456382 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:33.456387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:33.456390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:33.456425 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:33.458852 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:33.459396 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:33.459399 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:33.459401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:33.459512 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:33.459778 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:33.459781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:33.459783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:33.460024 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:33.460192 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:33.460400 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 14:31:33.460552 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:31:33.464380 9285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:31:33.471055 9285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:31:33.493106 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.482403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41438 (local address 127.6.46.195:42153)
0810 14:31:33.483562 (+ 1159us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:33.483571 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:33.484101 (+ 530us) server_negotiation.cc:407] Connection header received
0810 14:31:33.484425 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:33.484434 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:33.484663 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:33.484837 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:33.485923 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.487567 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.490209 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.491640 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.491740 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.491958 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:33.492610 (+ 652us) server_negotiation.cc:299] Negotiation successful
0810 14:31:33.492768 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":996,"thread_start_us":812,"threads_started":1}
I20260810 14:31:33.493958 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.482121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:42153 (local address 127.0.0.1:41438)
0810 14:31:33.483941 (+ 1820us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:33.483995 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:33.484186 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:33.485174 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:33.485181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:33.485225 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:33.485671 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.485684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.487749 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.487754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.489998 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.490028 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.492137 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.492143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.492353 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.492403 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 14:31:33.493395 (+ 992us) client_negotiation.cc:240] Negotiation successful
0810 14:31:33.493616 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1579,"thread_start_us":1612,"threads_started":1}
I20260810 14:31:33.496296 10159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.482209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:34451 (local address 127.0.0.1:42724)
0810 14:31:33.483979 (+ 1770us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:33.483993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:33.484163 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:33.486362 (+ 2199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:33.486367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:33.486391 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:33.486799 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.486811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.489822 (+ 3011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.489832 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.493872 (+ 4040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.493887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.495585 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.495591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.495748 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.495776 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 14:31:33.495853 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:31:33.495971 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1654,"thread_start_us":1364,"threads_started":1}
I20260810 14:31:33.496323 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.482786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42724 (local address 127.6.46.193:34451)
0810 14:31:33.484713 (+ 1927us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:33.484721 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:33.484747 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:31:33.484845 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:33.484852 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:33.486150 (+ 1298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:33.486972 (+ 822us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:33.487173 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.489665 (+ 2492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.494039 (+ 4374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.495395 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.495514 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.495714 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:33.495913 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 14:31:33.496063 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1751,"thread_start_us":1616,"threads_started":1}
I20260810 14:31:33.497845 10011 tablet_service.cc:1467] Processing CreateTablet for tablet 6e63285b2f2042a394f7b0ac59299c95 (DEFAULT_TABLE table=test-table [id=b88be2974df14e5c836a05a730b30249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:33.499404 10011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e63285b2f2042a394f7b0ac59299c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:33.499965 9516 tablet_service.cc:1467] Processing CreateTablet for tablet 6e63285b2f2042a394f7b0ac59299c95 (DEFAULT_TABLE table=test-table [id=b88be2974df14e5c836a05a730b30249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:33.500725 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.485091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43526 (local address 127.6.46.194:42141)
0810 14:31:33.493796 (+ 8705us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:33.493804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:33.493830 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:31:33.493932 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:33.493939 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:33.494180 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:33.494932 (+ 752us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:33.495106 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.496444 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.498919 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.499804 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.499928 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.500108 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:33.500316 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 14:31:33.500471 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8569,"thread_start_us":8423,"threads_started":1}
I20260810 14:31:33.500749 10163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.484849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:42141 (local address 127.0.0.1:43526)
0810 14:31:33.493043 (+ 8194us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:33.493062 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:33.493183 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:33.494410 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:33.494414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:33.494441 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:33.494741 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.494753 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.496641 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.496646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.498787 (+ 2141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.498801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.499970 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.499976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.500133 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.500164 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 14:31:33.500246 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 14:31:33.500357 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8045,"thread_start_us":8381,"threads_started":1}
I20260810 14:31:33.501297 9516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e63285b2f2042a394f7b0ac59299c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:33.504650 9766 tablet_service.cc:1467] Processing CreateTablet for tablet 6e63285b2f2042a394f7b0ac59299c95 (DEFAULT_TABLE table=test-table [id=b88be2974df14e5c836a05a730b30249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:33.505635 9766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e63285b2f2042a394f7b0ac59299c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:33.514788 10167 tablet_bootstrap.cc:492] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Bootstrap starting.
I20260810 14:31:33.516904 10171 tablet_bootstrap.cc:492] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0: Bootstrap starting.
I20260810 14:31:33.518347 10167 tablet_bootstrap.cc:654] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:33.518680 10169 tablet_bootstrap.cc:492] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939: Bootstrap starting.
I20260810 14:31:33.520254 10167 log.cc:824] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:33.520406 10171 tablet_bootstrap.cc:654] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:33.521708 10171 log.cc:824] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:33.521793 10169 tablet_bootstrap.cc:654] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:33.523020 10169 log.cc:824] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:33.523693 10167 tablet_bootstrap.cc:492] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: No bootstrap required, opened a new log
I20260810 14:31:33.523934 10167 ts_tablet_manager.cc:1397] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 14:31:33.524437 10171 tablet_bootstrap.cc:492] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0: No bootstrap required, opened a new log
I20260810 14:31:33.524672 10171 ts_tablet_manager.cc:1397] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260810 14:31:33.525827 10169 tablet_bootstrap.cc:492] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939: No bootstrap required, opened a new log
I20260810 14:31:33.526115 10169 ts_tablet_manager.cc:1397] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 14:31:33.532009 10171 raft_consensus.cc:348] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } }
I20260810 14:31:33.532328 10171 raft_consensus.cc:374] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:33.532428 10171 raft_consensus.cc:729] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e9412ec942418d8aa01581235ffbd0, State: Initialized, Role: FOLLOWER
I20260810 14:31:33.532965 10171 consensus_queue.cc:260] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } }
I20260810 14:31:33.533892 9883 heartbeater.cc:503] Master 127.6.46.254:34309 was elected leader, sending a full tablet report...
I20260810 14:31:33.533993 10167 raft_consensus.cc:348] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } }
I20260810 14:31:33.534397 10167 raft_consensus.cc:374] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:33.534487 10167 raft_consensus.cc:729] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95502ddb7e9f4ab493cf1ef1e3ea7e4a, State: Initialized, Role: FOLLOWER
I20260810 14:31:33.534562 10171 ts_tablet_manager.cc:1428] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260810 14:31:33.535120 10167 consensus_queue.cc:260] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } }
I20260810 14:31:33.536145 10132 heartbeater.cc:503] Master 127.6.46.254:34309 was elected leader, sending a full tablet report...
I20260810 14:31:33.536408 10169 raft_consensus.cc:348] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } }
I20260810 14:31:33.536794 10169 raft_consensus.cc:374] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:33.536867 10167 ts_tablet_manager.cc:1428] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260810 14:31:33.536885 10169 raft_consensus.cc:729] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd697cce9db247adb522296cbbf1d939, State: Initialized, Role: FOLLOWER
I20260810 14:31:33.537420 10169 consensus_queue.cc:260] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } }
I20260810 14:31:33.539016 10169 ts_tablet_manager.cc:1428] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260810 14:31:33.655067 10134 tablet.cc:2369] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:31:33.663659 9886 tablet.cc:2369] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:31:33.678439 9652 tablet.cc:2369] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:31:33.679855 10177 raft_consensus.cc:482] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:31:33.680171 10177 raft_consensus.cc:504] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } }
I20260810 14:31:33.681813 10177 leader_election.cc:288] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153), bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451)
I20260810 14:31:33.688591 10162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.682362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:40437 (local address 127.6.46.193:34451)
0810 14:31:33.682559 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:33.682565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:33.683201 (+ 636us) server_negotiation.cc:407] Connection header received
0810 14:31:33.683362 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:33.683366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:33.683424 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:33.683500 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:33.684168 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.684974 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.686960 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.687703 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.687865 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.687936 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:33.688244 (+ 308us) server_negotiation.cc:299] Negotiation successful
0810 14:31:33.688363 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:31:33.688651 10161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.682345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:42861 (local address 127.6.46.195:42153)
0810 14:31:33.682546 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:33.682554 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:33.683171 (+ 617us) server_negotiation.cc:407] Connection header received
0810 14:31:33.683351 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:33.683356 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:33.683418 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:33.683490 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:33.684079 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.684844 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.686877 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.687675 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.687814 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.687888 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:33.688263 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 14:31:33.688390 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:31:33.688767 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.682145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:42153 (local address 127.6.46.194:42861)
0810 14:31:33.683047 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:33.683066 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:33.683211 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:33.683520 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:33.683524 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:33.683548 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:33.683940 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.683953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.684969 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.684974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.686727 (+ 1753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.686740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.687897 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.687903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.688071 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.688094 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:31:33.688172 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:31:33.688290 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":49,"thread_start_us":428,"threads_started":1}
I20260810 14:31:33.688844 10181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.682254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:34451 (local address 127.6.46.194:40437)
0810 14:31:33.683094 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:33.683108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:33.683240 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:33.683530 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:33.683534 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:33.683576 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:33.684007 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.684020 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.685154 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.685158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.686799 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.686815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.687927 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.687934 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.688058 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.688089 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 14:31:33.688204 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 14:31:33.688395 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":21,"spinlock_wait_cycles":1408,"thread_start_us":547,"threads_started":1}
I20260810 14:31:33.689962 9563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e63285b2f2042a394f7b0ac59299c95" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd697cce9db247adb522296cbbf1d939" is_pre_election: true
I20260810 14:31:33.689962 10044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e63285b2f2042a394f7b0ac59299c95" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" is_pre_election: true
I20260810 14:31:33.690564 9563 raft_consensus.cc:2454] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 0.
I20260810 14:31:33.690564 10044 raft_consensus.cc:2454] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 0.
I20260810 14:31:33.691277 9684 leader_election.cc:302] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd697cce9db247adb522296cbbf1d939, f3e9412ec942418d8aa01581235ffbd0; no voters:
I20260810 14:31:33.691674 10177 raft_consensus.cc:2781] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:31:33.691833 10177 raft_consensus.cc:482] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:31:33.691920 10177 raft_consensus.cc:3037] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:33.694084 10177 raft_consensus.cc:504] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } }
I20260810 14:31:33.694639 10177 leader_election.cc:288] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 election: Requested vote from peers 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153), bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451)
I20260810 14:31:33.695089 9563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e63285b2f2042a394f7b0ac59299c95" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd697cce9db247adb522296cbbf1d939"
I20260810 14:31:33.695065 10044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e63285b2f2042a394f7b0ac59299c95" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a"
I20260810 14:31:33.695269 9563 raft_consensus.cc:3037] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:33.695299 10044 raft_consensus.cc:3037] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:33.697407 9563 raft_consensus.cc:2454] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 1.
I20260810 14:31:33.697475 10044 raft_consensus.cc:2454] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 1.
I20260810 14:31:33.697746 9684 leader_election.cc:302] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd697cce9db247adb522296cbbf1d939, f3e9412ec942418d8aa01581235ffbd0; no voters:
I20260810 14:31:33.698035 10177 raft_consensus.cc:2781] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:33.698491 10177 raft_consensus.cc:686] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [term 1 LEADER]: Becoming Leader. State: Replica: f3e9412ec942418d8aa01581235ffbd0, State: Running, Role: LEADER
I20260810 14:31:33.698935 10177 consensus_queue.cc:237] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } }
I20260810 14:31:33.703051 9282 catalog_manager.cc:5495] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3e9412ec942418d8aa01581235ffbd0 (127.6.46.194). New cstate: current_term: 1 leader_uuid: "f3e9412ec942418d8aa01581235ffbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } health_report { overall_health: HEALTHY } } }
Aug 10 14:31:33 dist-test-slave-q9h9 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372291, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:33.752995 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.740301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:34309 (local address 127.0.0.1:48426)
0810 14:31:33.740874 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:33.740894 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:33.741017 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:33.741314 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:33.741317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:33.741960 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:33.742271 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.742281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.743289 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.743293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.744280 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.744291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.745030 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.745034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.745168 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.745800 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:33.745817 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:33.748072 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:33.750783 (+ 2711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:33.750796 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:33.750802 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:33.751257 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:33.751564 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:33.751567 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:33.751569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:33.751707 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:33.752142 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:33.752148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:33.752291 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 14:31:33.752420 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 14:31:33.752618 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":317,"threads_started":1}
I20260810 14:31:33.753028 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.740239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48426 (local address 127.6.46.254:34309)
0810 14:31:33.740416 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:33.740422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:33.740973 (+ 551us) server_negotiation.cc:407] Connection header received
0810 14:31:33.741121 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:33.741125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:33.741186 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:33.741256 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:33.742401 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.743147 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.744424 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.744872 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.744969 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.748236 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:33.748241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:33.748244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:33.748273 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:33.750634 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:33.751354 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:33.751357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:33.751360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:33.751477 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:33.751789 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:33.751792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:33.751794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:33.752041 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:33.752144 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:33.752508 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 14:31:33.752743 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":50}
I20260810 14:31:33.757602 9282 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:48426:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 14:31:33.758622 9282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:31:33.764807 9516 tablet_service.cc:1467] Processing CreateTablet for tablet f3d26b1f40eb472b8decafe3123bf414 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=81eac815e4464fabbdfd024ae089a0cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:31:33.764845 9766 tablet_service.cc:1467] Processing CreateTablet for tablet f3d26b1f40eb472b8decafe3123bf414 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=81eac815e4464fabbdfd024ae089a0cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:31:33.765183 10011 tablet_service.cc:1467] Processing CreateTablet for tablet f3d26b1f40eb472b8decafe3123bf414 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=81eac815e4464fabbdfd024ae089a0cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:31:33.765776 9766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d26b1f40eb472b8decafe3123bf414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:33.765777 9516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d26b1f40eb472b8decafe3123bf414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:33.765945 10011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3d26b1f40eb472b8decafe3123bf414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:33.771667 10167 tablet_bootstrap.cc:492] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Bootstrap starting.
I20260810 14:31:33.772728 10171 tablet_bootstrap.cc:492] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0: Bootstrap starting.
I20260810 14:31:33.773277 10169 tablet_bootstrap.cc:492] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939: Bootstrap starting.
I20260810 14:31:33.773340 10167 tablet_bootstrap.cc:654] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:33.774380 10171 tablet_bootstrap.cc:654] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:33.774884 10169 tablet_bootstrap.cc:654] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:33.775772 10167 tablet_bootstrap.cc:492] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: No bootstrap required, opened a new log
I20260810 14:31:33.775902 10167 ts_tablet_manager.cc:1397] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260810 14:31:33.776503 10167 raft_consensus.cc:348] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:33.776693 10167 raft_consensus.cc:374] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:33.776780 10167 raft_consensus.cc:729] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95502ddb7e9f4ab493cf1ef1e3ea7e4a, State: Initialized, Role: FOLLOWER
I20260810 14:31:33.777009 10167 consensus_queue.cc:260] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:33.777415 10178 tablet_replica.cc:439] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } }
I20260810 14:31:33.777804 10178 tablet_replica.cc:442] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:33.778661 10167 ts_tablet_manager.cc:1428] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 14:31:33.780313 10169 tablet_bootstrap.cc:492] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939: No bootstrap required, opened a new log
I20260810 14:31:33.780463 10169 ts_tablet_manager.cc:1397] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260810 14:31:33.780614 10171 tablet_bootstrap.cc:492] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0: No bootstrap required, opened a new log
I20260810 14:31:33.780758 10171 ts_tablet_manager.cc:1397] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260810 14:31:33.781214 10169 raft_consensus.cc:348] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:33.781452 10169 raft_consensus.cc:374] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:33.781551 10169 raft_consensus.cc:729] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd697cce9db247adb522296cbbf1d939, State: Initialized, Role: FOLLOWER
I20260810 14:31:33.781493 10171 raft_consensus.cc:348] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:33.781719 10171 raft_consensus.cc:374] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:33.781814 10171 raft_consensus.cc:729] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e9412ec942418d8aa01581235ffbd0, State: Initialized, Role: FOLLOWER
I20260810 14:31:33.781793 10169 consensus_queue.cc:260] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:33.782050 10171 consensus_queue.cc:260] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:33.782439 10169 ts_tablet_manager.cc:1428] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 14:31:33.782316 10179 tablet_replica.cc:439] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } }
I20260810 14:31:33.782471 10177 tablet_replica.cc:439] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } }
I20260810 14:31:33.782706 10179 tablet_replica.cc:442] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:33.782728 10177 tablet_replica.cc:442] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:33.783176 10171 ts_tablet_manager.cc:1428] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 14:31:33.871634 10177 raft_consensus.cc:482] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:31:33.871838 10177 raft_consensus.cc:504] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:33.872506 10177 leader_election.cc:288] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451), 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153)
I20260810 14:31:33.873003 9563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d26b1f40eb472b8decafe3123bf414" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd697cce9db247adb522296cbbf1d939" is_pre_election: true
I20260810 14:31:33.873077 10044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d26b1f40eb472b8decafe3123bf414" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" is_pre_election: true
I20260810 14:31:33.873237 9563 raft_consensus.cc:2454] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 0.
I20260810 14:31:33.873363 10044 raft_consensus.cc:2454] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 0.
I20260810 14:31:33.873673 9684 leader_election.cc:302] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd697cce9db247adb522296cbbf1d939, f3e9412ec942418d8aa01581235ffbd0; no voters:
I20260810 14:31:33.873940 10177 raft_consensus.cc:2781] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:31:33.874048 10177 raft_consensus.cc:482] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:31:33.874130 10177 raft_consensus.cc:3037] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:33.876112 10177 raft_consensus.cc:504] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:33.876590 10177 leader_election.cc:288] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 election: Requested vote from peers bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451), 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153)
I20260810 14:31:33.876997 10044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d26b1f40eb472b8decafe3123bf414" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a"
I20260810 14:31:33.877000 9563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3d26b1f40eb472b8decafe3123bf414" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd697cce9db247adb522296cbbf1d939"
I20260810 14:31:33.877189 9563 raft_consensus.cc:3037] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:33.877199 10044 raft_consensus.cc:3037] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:33.879171 9563 raft_consensus.cc:2454] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 1.
I20260810 14:31:33.879184 10044 raft_consensus.cc:2454] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 1.
I20260810 14:31:33.879521 9684 leader_election.cc:302] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd697cce9db247adb522296cbbf1d939, f3e9412ec942418d8aa01581235ffbd0; no voters:
I20260810 14:31:33.879752 10177 raft_consensus.cc:2781] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:33.879914 10177 raft_consensus.cc:686] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [term 1 LEADER]: Becoming Leader. State: Replica: f3e9412ec942418d8aa01581235ffbd0, State: Running, Role: LEADER
I20260810 14:31:33.880155 10177 consensus_queue.cc:237] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:33.881744 10183 tablet_replica.cc:439] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0: TxnStatusTablet state changed. Reason: New leader f3e9412ec942418d8aa01581235ffbd0. Latest consensus state: current_term: 1 leader_uuid: "f3e9412ec942418d8aa01581235ffbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } }
I20260810 14:31:33.882157 10183 tablet_replica.cc:442] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:31:33.882803 10198 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 14:31:33.883054 10198 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 14:31:33.882946 9284 catalog_manager.cc:5495] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3e9412ec942418d8aa01581235ffbd0 (127.6.46.194). New cstate: current_term: 1 leader_uuid: "f3e9412ec942418d8aa01581235ffbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } health_report { overall_health: UNKNOWN } } }
I20260810 14:31:33.901157 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.894991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43532 (local address 127.6.46.194:42141)
0810 14:31:33.895212 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:33.895216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:33.895295 (+ 79us) server_negotiation.cc:407] Connection header received
0810 14:31:33.895593 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:33.895597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:33.895671 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:33.895775 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:31:33.896220 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.896995 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.898955 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.899413 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.899516 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.900090 (+ 574us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:31:33.900627 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:31:33.900727 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:33.900872 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 14:31:33.900935 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 14:31:33.901307 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.894909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:42141 (local address 127.0.0.1:43532)
0810 14:31:33.895214 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:33.895234 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:33.895383 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:33.895825 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:33.895830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:33.895842 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:31:33.896048 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.896059 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.897138 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.897144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.898813 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.898829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.899563 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.899570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.899798 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.899871 (+ 73us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:31:33.900735 (+ 864us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:31:33.900757 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:31:33.900850 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 14:31:33.901002 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 14:31:33.909365 10044 raft_consensus.cc:1261] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Refusing update from remote peer f3e9412ec942418d8aa01581235ffbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:33.909444 9563 raft_consensus.cc:1261] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Refusing update from remote peer f3e9412ec942418d8aa01581235ffbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:33.910493 10183 consensus_queue.cc:1035] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:33.911000 10177 consensus_queue.cc:1035] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:33.932960 10179 tablet_replica.cc:439] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939: TxnStatusTablet state changed. Reason: New leader f3e9412ec942418d8aa01581235ffbd0. Latest consensus state: current_term: 1 leader_uuid: "f3e9412ec942418d8aa01581235ffbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } }
I20260810 14:31:33.933254 10179 tablet_replica.cc:442] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:33.936780 10178 tablet_replica.cc:439] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: TxnStatusTablet state changed. Reason: New leader f3e9412ec942418d8aa01581235ffbd0. Latest consensus state: current_term: 1 leader_uuid: "f3e9412ec942418d8aa01581235ffbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } }
I20260810 14:31:33.937033 10178 tablet_replica.cc:442] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:33.939170 10203 mvcc.cc:205] Tried to move back new op lower bound from 7316980915843346432 to 7316980915735212032. Current Snapshot: MvccSnapshot[applied={T|T < 7316980915843346432}]
I20260810 14:31:33.945894 10178 tablet_replica.cc:439] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3e9412ec942418d8aa01581235ffbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } }
I20260810 14:31:33.946159 10178 tablet_replica.cc:442] T f3d26b1f40eb472b8decafe3123bf414 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:33.951095 10177 tablet_replica.cc:439] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3e9412ec942418d8aa01581235ffbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } }
I20260810 14:31:33.951498 10177 tablet_replica.cc:442] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:31:33.951804 10177 tablet_replica.cc:439] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3e9412ec942418d8aa01581235ffbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } }
I20260810 14:31:33.952131 10177 tablet_replica.cc:442] T f3d26b1f40eb472b8decafe3123bf414 P f3e9412ec942418d8aa01581235ffbd0: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:31:33.955240 10179 tablet_replica.cc:439] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3e9412ec942418d8aa01581235ffbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } } }
I20260810 14:31:33.955470 10179 tablet_replica.cc:442] T f3d26b1f40eb472b8decafe3123bf414 P bd697cce9db247adb522296cbbf1d939: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:31:33.963197 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.957809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43546 (local address 127.6.46.194:42141)
0810 14:31:33.957971 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:33.957977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:33.958213 (+ 236us) server_negotiation.cc:407] Connection header received
0810 14:31:33.958338 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:33.958344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:33.958434 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:33.958530 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:31:33.958821 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.960026 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.961312 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.961989 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.962147 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.962488 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:31:33.962643 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:31:33.962739 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:33.962818 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 14:31:33.962895 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:31:33.980127 10166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.973912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:60163 (local address 127.6.46.194:42141)
0810 14:31:33.974127 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:33.974133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:33.974791 (+ 658us) server_negotiation.cc:407] Connection header received
0810 14:31:33.974931 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:33.974937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:33.975015 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:33.975117 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:31:33.975611 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.976688 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.978161 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.978855 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.978976 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.979402 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:31:33.979562 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:31:33.979650 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:33.979753 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 14:31:33.979829 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:31:33.980139 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:33.973949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:42141 (local address 127.6.46.194:60163)
0810 14:31:33.974686 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:33.974702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:33.974827 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:33.975132 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:33.975136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:33.975145 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:31:33.975458 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.975465 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.976822 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.976825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.977997 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:33.978048 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:33.979014 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:33.979018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:33.979158 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:33.979212 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:31:33.979652 (+ 440us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:31:33.979660 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 14:31:33.979715 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 14:31:33.979812 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":428,"threads_started":1}
I20260810 14:31:33.989357 9562 raft_consensus.cc:1261] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Refusing update from remote peer f3e9412ec942418d8aa01581235ffbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:33.989431 10044 raft_consensus.cc:1261] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Refusing update from remote peer f3e9412ec942418d8aa01581235ffbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:33.989925 10183 consensus_queue.cc:1035] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:33.990278 10177 consensus_queue.cc:1035] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:33.994675 10201 mvcc.cc:205] Tried to move back new op lower bound from 7316980916175994880 to 7316980914994282496. Current Snapshot: MvccSnapshot[applied={T|T < 7316980916175994880}]
I20260810 14:31:34.055332 9285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:34.055577 9285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:34.058750 9285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744: Sending DeleteTablet for 3 replicas of tablet 6e63285b2f2042a394f7b0ac59299c95
I20260810 14:31:34.059772 10011 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e63285b2f2042a394f7b0ac59299c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:34 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:41438
I20260810 14:31:34.059901 9766 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e63285b2f2042a394f7b0ac59299c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:34 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:43526
I20260810 14:31:34.060527 9516 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e63285b2f2042a394f7b0ac59299c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:34 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:42724
I20260810 14:31:34.061551 9285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48420:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:31:34.062361 9285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:31:34.064462 10230 tablet_replica.cc:330] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: stopping tablet replica
I20260810 14:31:34.064958 10230 raft_consensus.cc:2229] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:34.065399 10230 raft_consensus.cc:2258] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:34.067381 10230 ts_tablet_manager.cc:1905] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:34.070731 9514 tablet_service.cc:1467] Processing CreateTablet for tablet deb08b430849427bb59e5a68506ec66e (DEFAULT_TABLE table=test-table [id=b7e848de2b974c60931083b026fe41c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:34.071290 9514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb08b430849427bb59e5a68506ec66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:34.071846 10231 tablet_replica.cc:330] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0: stopping tablet replica
I20260810 14:31:34.073868 10231 raft_consensus.cc:2229] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:31:34.074611 10011 tablet_service.cc:1467] Processing CreateTablet for tablet deb08b430849427bb59e5a68506ec66e (DEFAULT_TABLE table=test-table [id=b7e848de2b974c60931083b026fe41c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:34.075090 10011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb08b430849427bb59e5a68506ec66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:34.078809 9765 tablet_service.cc:1467] Processing CreateTablet for tablet deb08b430849427bb59e5a68506ec66e (DEFAULT_TABLE table=test-table [id=b7e848de2b974c60931083b026fe41c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:34.080466 9765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet deb08b430849427bb59e5a68506ec66e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:34.083426 10169 tablet_bootstrap.cc:492] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939: Bootstrap starting.
I20260810 14:31:34.085016 10169 tablet_bootstrap.cc:654] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:34.088064 10167 tablet_bootstrap.cc:492] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Bootstrap starting.
I20260810 14:31:34.089366 10171 tablet_bootstrap.cc:492] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0: Bootstrap starting.
I20260810 14:31:34.089779 10167 tablet_bootstrap.cc:654] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:34.091046 10171 tablet_bootstrap.cc:654] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:34.075474 10231 raft_consensus.cc:2258] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:34.099285 10231 ts_tablet_manager.cc:1905] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:34.100695 10233 tablet_replica.cc:330] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939: stopping tablet replica
I20260810 14:31:34.101194 10233 raft_consensus.cc:2229] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:34.101639 10233 raft_consensus.cc:2258] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:34.102656 10167 tablet_bootstrap.cc:492] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: No bootstrap required, opened a new log
I20260810 14:31:34.102684 10169 tablet_bootstrap.cc:492] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939: No bootstrap required, opened a new log
I20260810 14:31:34.102829 10167 ts_tablet_manager.cc:1397] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20260810 14:31:34.102829 10169 ts_tablet_manager.cc:1397] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.012s
I20260810 14:31:34.103561 10169 raft_consensus.cc:348] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:34.103590 10167 raft_consensus.cc:348] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:34.103814 10169 raft_consensus.cc:374] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:34.103823 10167 raft_consensus.cc:374] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:34.103897 10167 raft_consensus.cc:729] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95502ddb7e9f4ab493cf1ef1e3ea7e4a, State: Initialized, Role: FOLLOWER
I20260810 14:31:34.103897 10169 raft_consensus.cc:729] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd697cce9db247adb522296cbbf1d939, State: Initialized, Role: FOLLOWER
I20260810 14:31:34.104168 10169 consensus_queue.cc:260] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:34.104163 10167 consensus_queue.cc:260] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:34.105432 10169 ts_tablet_manager.cc:1428] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 14:31:34.106909 10233 ts_tablet_manager.cc:1905] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:34.108165 10167 ts_tablet_manager.cc:1428] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260810 14:31:34.110987 10171 tablet_bootstrap.cc:492] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0: No bootstrap required, opened a new log
I20260810 14:31:34.111171 10171 ts_tablet_manager.cc:1397] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.007s
I20260810 14:31:34.111961 10171 raft_consensus.cc:348] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:34.112218 10171 raft_consensus.cc:374] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:34.112303 10171 raft_consensus.cc:729] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e9412ec942418d8aa01581235ffbd0, State: Initialized, Role: FOLLOWER
I20260810 14:31:34.112578 10171 consensus_queue.cc:260] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:34.113287 10171 ts_tablet_manager.cc:1428] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 14:31:34.113934 10230 ts_tablet_manager.cc:1918] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:31:34.115087 10233 ts_tablet_manager.cc:1918] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:31:34.115280 10233 log.cc:1196] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/6e63285b2f2042a394f7b0ac59299c95
I20260810 14:31:34.115834 10233 ts_tablet_manager.cc:1939] T 6e63285b2f2042a394f7b0ac59299c95 P bd697cce9db247adb522296cbbf1d939: Deleting consensus metadata
I20260810 14:31:34.117542 9259 catalog_manager.cc:4841] TS bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451): tablet 6e63285b2f2042a394f7b0ac59299c95 (table test-table [id=b88be2974df14e5c836a05a730b30249]) successfully deleted
I20260810 14:31:34.118170 10230 log.cc:1196] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/6e63285b2f2042a394f7b0ac59299c95
I20260810 14:31:34.118662 10230 ts_tablet_manager.cc:1939] T 6e63285b2f2042a394f7b0ac59299c95 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Deleting consensus metadata
I20260810 14:31:34.118834 10231 ts_tablet_manager.cc:1918] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:31:34.118990 10231 log.cc:1196] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/6e63285b2f2042a394f7b0ac59299c95
I20260810 14:31:34.119558 10231 ts_tablet_manager.cc:1939] T 6e63285b2f2042a394f7b0ac59299c95 P f3e9412ec942418d8aa01581235ffbd0: Deleting consensus metadata
I20260810 14:31:34.120311 9257 catalog_manager.cc:4841] TS 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153): tablet 6e63285b2f2042a394f7b0ac59299c95 (table test-table [id=b88be2974df14e5c836a05a730b30249]) successfully deleted
I20260810 14:31:34.121037 9257 catalog_manager.cc:4841] TS f3e9412ec942418d8aa01581235ffbd0 (127.6.46.194:42141): tablet 6e63285b2f2042a394f7b0ac59299c95 (table test-table [id=b88be2974df14e5c836a05a730b30249]) successfully deleted
I20260810 14:31:34.180043 10224 raft_consensus.cc:482] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:31:34.180262 10224 raft_consensus.cc:504] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:34.180990 10224 leader_election.cc:288] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451), 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153)
I20260810 14:31:34.181442 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb08b430849427bb59e5a68506ec66e" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd697cce9db247adb522296cbbf1d939" is_pre_election: true
I20260810 14:31:34.181766 9562 raft_consensus.cc:2454] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 0.
I20260810 14:31:34.181711 10044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb08b430849427bb59e5a68506ec66e" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" is_pre_election: true
I20260810 14:31:34.181943 10044 raft_consensus.cc:2454] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 0.
I20260810 14:31:34.182286 9684 leader_election.cc:302] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd697cce9db247adb522296cbbf1d939, f3e9412ec942418d8aa01581235ffbd0; no voters:
I20260810 14:31:34.182528 10224 raft_consensus.cc:2781] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:31:34.182655 10224 raft_consensus.cc:482] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:31:34.182756 10224 raft_consensus.cc:3037] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:34.185025 10224 raft_consensus.cc:504] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:34.185580 10224 leader_election.cc:288] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 election: Requested vote from peers bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451), 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153)
I20260810 14:31:34.186159 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb08b430849427bb59e5a68506ec66e" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd697cce9db247adb522296cbbf1d939"
I20260810 14:31:34.186163 10044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "deb08b430849427bb59e5a68506ec66e" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a"
I20260810 14:31:34.186348 9562 raft_consensus.cc:3037] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:34.186416 10044 raft_consensus.cc:3037] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:34.188359 9562 raft_consensus.cc:2454] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 1.
I20260810 14:31:34.188724 9684 leader_election.cc:302] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd697cce9db247adb522296cbbf1d939, f3e9412ec942418d8aa01581235ffbd0; no voters:
I20260810 14:31:34.188979 10224 raft_consensus.cc:2781] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:34.189136 10224 raft_consensus.cc:686] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [term 1 LEADER]: Becoming Leader. State: Replica: f3e9412ec942418d8aa01581235ffbd0, State: Running, Role: LEADER
I20260810 14:31:34.189482 10224 consensus_queue.cc:237] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:34.189970 10044 raft_consensus.cc:2454] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 1.
I20260810 14:31:34.192325 9284 catalog_manager.cc:5495] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3e9412ec942418d8aa01581235ffbd0 (127.6.46.194). New cstate: current_term: 1 leader_uuid: "f3e9412ec942418d8aa01581235ffbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } health_report { overall_health: UNKNOWN } } }
I20260810 14:31:34.243156 10044 raft_consensus.cc:1261] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Refusing update from remote peer f3e9412ec942418d8aa01581235ffbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:34.243230 9562 raft_consensus.cc:1261] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Refusing update from remote peer f3e9412ec942418d8aa01581235ffbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:34.243747 10177 consensus_queue.cc:1035] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:34.244009 10224 consensus_queue.cc:1035] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:31:34.270550 10201 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 14:31:35.850057 9285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:35.850282 9285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48420:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:35.854396 9285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744: Sending DeleteTablet for 3 replicas of tablet deb08b430849427bb59e5a68506ec66e
I20260810 14:31:35.855186 9516 tablet_service.cc:1514] Processing DeleteTablet for tablet deb08b430849427bb59e5a68506ec66e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:35 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:42724
I20260810 14:31:35.855388 9765 tablet_service.cc:1514] Processing DeleteTablet for tablet deb08b430849427bb59e5a68506ec66e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:35 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:43526
I20260810 14:31:35.858207 10011 tablet_service.cc:1514] Processing DeleteTablet for tablet deb08b430849427bb59e5a68506ec66e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:35 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:41438
I20260810 14:31:35.866556 10274 tablet_replica.cc:330] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0: stopping tablet replica
I20260810 14:31:35.866741 10274 raft_consensus.cc:2229] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:31:35.867063 10274 raft_consensus.cc:2258] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:35.870303 10274 ts_tablet_manager.cc:1905] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:35.874550 10275 tablet_replica.cc:330] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: stopping tablet replica
I20260810 14:31:35.874770 10275 raft_consensus.cc:2229] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:35.875042 10275 raft_consensus.cc:2258] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:35.875993 10274 ts_tablet_manager.cc:1918] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:31:35.876133 10274 log.cc:1196] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/deb08b430849427bb59e5a68506ec66e
I20260810 14:31:35.876590 10274 ts_tablet_manager.cc:1939] T deb08b430849427bb59e5a68506ec66e P f3e9412ec942418d8aa01581235ffbd0: Deleting consensus metadata
I20260810 14:31:35.877287 10275 ts_tablet_manager.cc:1905] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:35.877693 9257 catalog_manager.cc:4841] TS f3e9412ec942418d8aa01581235ffbd0 (127.6.46.194:42141): tablet deb08b430849427bb59e5a68506ec66e (table test-table [id=b7e848de2b974c60931083b026fe41c9]) successfully deleted
I20260810 14:31:35.878369 10273 tablet_replica.cc:330] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939: stopping tablet replica
I20260810 14:31:35.878561 10273 raft_consensus.cc:2229] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:35.878800 10273 raft_consensus.cc:2258] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:35.879629 10273 ts_tablet_manager.cc:1905] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:35.883395 10275 ts_tablet_manager.cc:1918] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:31:35.883526 10275 log.cc:1196] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/deb08b430849427bb59e5a68506ec66e
I20260810 14:31:35.883963 10275 ts_tablet_manager.cc:1939] T deb08b430849427bb59e5a68506ec66e P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Deleting consensus metadata
I20260810 14:31:35.885059 9257 catalog_manager.cc:4841] TS 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153): tablet deb08b430849427bb59e5a68506ec66e (table test-table [id=b7e848de2b974c60931083b026fe41c9]) successfully deleted
I20260810 14:31:35.885706 10273 ts_tablet_manager.cc:1918] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:31:35.885828 10273 log.cc:1196] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/deb08b430849427bb59e5a68506ec66e
I20260810 14:31:35.886312 10273 ts_tablet_manager.cc:1939] T deb08b430849427bb59e5a68506ec66e P bd697cce9db247adb522296cbbf1d939: Deleting consensus metadata
I20260810 14:31:35.887203 9259 catalog_manager.cc:4841] TS bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451): tablet deb08b430849427bb59e5a68506ec66e (table test-table [id=b7e848de2b974c60931083b026fe41c9]) successfully deleted
Aug 10 14:31:35 dist-test-slave-q9h9 krb5kdc[9213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:31:35 dist-test-slave-q9h9 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:35.920362 10287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:35.905926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48436 (local address 127.6.46.254:34309)
0810 14:31:35.906604 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:35.906609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:35.906631 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:31:35.906789 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:35.906793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:35.906855 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:35.906942 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:35.908162 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:35.908924 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:35.910269 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:35.910682 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:35.910796 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:35.914429 (+ 3633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:35.914437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:35.914441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:35.914482 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:35.917699 (+ 3217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:35.918481 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:35.918487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:35.918491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:35.918665 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:35.918979 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:35.918983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:35.918987 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:35.919308 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:35.919522 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:35.919767 (+ 245us) server_negotiation.cc:299] Negotiation successful
0810 14:31:35.919978 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":418,"threads_started":1}
I20260810 14:31:35.924028 9285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:31:35.924711 9285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:31:35.931383 9516 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcdd4cbd7b34f1bb963ac15114a59ab (DEFAULT_TABLE table=test-table [id=805ddb79bd004838b2d547227dbc9153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:35.931701 9765 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcdd4cbd7b34f1bb963ac15114a59ab (DEFAULT_TABLE table=test-table [id=805ddb79bd004838b2d547227dbc9153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:35.931830 9516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcdd4cbd7b34f1bb963ac15114a59ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:35.931814 10011 tablet_service.cc:1467] Processing CreateTablet for tablet 5fcdd4cbd7b34f1bb963ac15114a59ab (DEFAULT_TABLE table=test-table [id=805ddb79bd004838b2d547227dbc9153]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:35.932178 10011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcdd4cbd7b34f1bb963ac15114a59ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:35.936964 10288 tablet_bootstrap.cc:492] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939: Bootstrap starting.
I20260810 14:31:35.937232 9765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fcdd4cbd7b34f1bb963ac15114a59ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:35.937992 10289 tablet_bootstrap.cc:492] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Bootstrap starting.
I20260810 14:31:35.938683 10288 tablet_bootstrap.cc:654] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:35.939517 10289 tablet_bootstrap.cc:654] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:35.944653 10289 tablet_bootstrap.cc:492] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: No bootstrap required, opened a new log
I20260810 14:31:35.944811 10289 ts_tablet_manager.cc:1397] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260810 14:31:35.945098 10288 tablet_bootstrap.cc:492] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939: No bootstrap required, opened a new log
I20260810 14:31:35.945286 10288 ts_tablet_manager.cc:1397] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20260810 14:31:35.945658 10289 raft_consensus.cc:348] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:35.945916 10289 raft_consensus.cc:374] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:35.946012 10289 raft_consensus.cc:729] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95502ddb7e9f4ab493cf1ef1e3ea7e4a, State: Initialized, Role: FOLLOWER
I20260810 14:31:35.946211 10288 raft_consensus.cc:348] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:35.946322 10289 consensus_queue.cc:260] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:35.946480 10288 raft_consensus.cc:374] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:35.946581 10288 raft_consensus.cc:729] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd697cce9db247adb522296cbbf1d939, State: Initialized, Role: FOLLOWER
I20260810 14:31:35.946817 10288 consensus_queue.cc:260] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:35.953183 10293 tablet_bootstrap.cc:492] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0: Bootstrap starting.
I20260810 14:31:35.953426 10289 ts_tablet_manager.cc:1428] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Time spent starting tablet: real 0.008s user 0.008s sys 0.001s
I20260810 14:31:35.954108 10288 ts_tablet_manager.cc:1428] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260810 14:31:35.956085 10293 tablet_bootstrap.cc:654] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:35.958231 10293 tablet_bootstrap.cc:492] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0: No bootstrap required, opened a new log
I20260810 14:31:35.958369 10293 ts_tablet_manager.cc:1397] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260810 14:31:35.959009 10293 raft_consensus.cc:348] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:35.959215 10293 raft_consensus.cc:374] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:35.959308 10293 raft_consensus.cc:729] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e9412ec942418d8aa01581235ffbd0, State: Initialized, Role: FOLLOWER
I20260810 14:31:35.959540 10293 consensus_queue.cc:260] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:35.960029 10293 ts_tablet_manager.cc:1428] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 14:31:36.079527 10224 raft_consensus.cc:482] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:31:36.079707 10224 raft_consensus.cc:504] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:36.080312 10224 leader_election.cc:288] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451), 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153)
I20260810 14:31:36.080806 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcdd4cbd7b34f1bb963ac15114a59ab" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd697cce9db247adb522296cbbf1d939" is_pre_election: true
I20260810 14:31:36.080917 10043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcdd4cbd7b34f1bb963ac15114a59ab" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" is_pre_election: true
I20260810 14:31:36.081069 9562 raft_consensus.cc:2454] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 0.
I20260810 14:31:36.081171 10043 raft_consensus.cc:2454] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 0.
I20260810 14:31:36.081597 9684 leader_election.cc:302] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd697cce9db247adb522296cbbf1d939, f3e9412ec942418d8aa01581235ffbd0; no voters:
I20260810 14:31:36.081890 10224 raft_consensus.cc:2781] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:31:36.082036 10224 raft_consensus.cc:482] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:31:36.082131 10224 raft_consensus.cc:3037] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:36.084180 10224 raft_consensus.cc:504] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:36.084626 10224 leader_election.cc:288] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 election: Requested vote from peers bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451), 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153)
I20260810 14:31:36.085052 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcdd4cbd7b34f1bb963ac15114a59ab" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd697cce9db247adb522296cbbf1d939"
I20260810 14:31:36.085058 10043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fcdd4cbd7b34f1bb963ac15114a59ab" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a"
I20260810 14:31:36.085217 10043 raft_consensus.cc:3037] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:36.085232 9562 raft_consensus.cc:3037] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:36.087211 9562 raft_consensus.cc:2454] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 1.
I20260810 14:31:36.087502 10043 raft_consensus.cc:2454] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 1.
I20260810 14:31:36.087544 9684 leader_election.cc:302] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd697cce9db247adb522296cbbf1d939, f3e9412ec942418d8aa01581235ffbd0; no voters:
I20260810 14:31:36.087821 10224 raft_consensus.cc:2781] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:36.087942 10224 raft_consensus.cc:686] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [term 1 LEADER]: Becoming Leader. State: Replica: f3e9412ec942418d8aa01581235ffbd0, State: Running, Role: LEADER
I20260810 14:31:36.088205 10224 consensus_queue.cc:237] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:36.090788 9285 catalog_manager.cc:5495] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3e9412ec942418d8aa01581235ffbd0 (127.6.46.194). New cstate: current_term: 1 leader_uuid: "f3e9412ec942418d8aa01581235ffbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } health_report { overall_health: UNKNOWN } } }
I20260810 14:31:36.136368 10306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:36.131254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.6.46.194:42141)
0810 14:31:36.131878 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:36.131882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:36.131899 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:31:36.131947 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:36.131950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:36.132008 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:36.132092 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:31:36.132509 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:36.133318 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:36.134671 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:36.135113 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:36.135227 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:36.135634 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:31:36.135750 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:31:36.135827 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:36.135971 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 14:31:36.136043 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":380,"threads_started":1}
I20260810 14:31:36.144212 10043 raft_consensus.cc:1261] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Refusing update from remote peer f3e9412ec942418d8aa01581235ffbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:36.144371 9562 raft_consensus.cc:1261] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Refusing update from remote peer f3e9412ec942418d8aa01581235ffbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:36.144697 10224 consensus_queue.cc:1035] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:36.144937 10249 consensus_queue.cc:1035] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:36.183670 9284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:36.183856 9284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:36.186383 9284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744: Sending DeleteTablet for 3 replicas of tablet 5fcdd4cbd7b34f1bb963ac15114a59ab
I20260810 14:31:36.187052 9516 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcdd4cbd7b34f1bb963ac15114a59ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:36 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:42724
I20260810 14:31:36.187068 9766 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcdd4cbd7b34f1bb963ac15114a59ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:36 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:43526
I20260810 14:31:36.187211 10011 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fcdd4cbd7b34f1bb963ac15114a59ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:36 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:41438
I20260810 14:31:36.187318 10274 tablet_replica.cc:330] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0: stopping tablet replica
I20260810 14:31:36.187466 10275 tablet_replica.cc:330] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: stopping tablet replica
I20260810 14:31:36.187515 10274 raft_consensus.cc:2229] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:31:36.187644 10275 raft_consensus.cc:2229] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:36.187798 10274 raft_consensus.cc:2258] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:36.187819 10275 raft_consensus.cc:2258] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:36.188071 10273 tablet_replica.cc:330] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939: stopping tablet replica
I20260810 14:31:36.188263 10273 raft_consensus.cc:2229] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:36.188436 10273 raft_consensus.cc:2258] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:36.188318 9284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:31:36.188987 9284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:31:36.189020 10275 ts_tablet_manager.cc:1905] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:36.189473 10274 ts_tablet_manager.cc:1905] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:36.195374 10011 tablet_service.cc:1467] Processing CreateTablet for tablet d25b17ece16940e8870521c5d1a31136 (DEFAULT_TABLE table=test-table [id=ac23e3f435bf4ff991f495d7747ee447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:36.195387 9516 tablet_service.cc:1467] Processing CreateTablet for tablet d25b17ece16940e8870521c5d1a31136 (DEFAULT_TABLE table=test-table [id=ac23e3f435bf4ff991f495d7747ee447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:36.195650 10274 ts_tablet_manager.cc:1918] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:31:36.195786 9516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25b17ece16940e8870521c5d1a31136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:36.195794 10011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25b17ece16940e8870521c5d1a31136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:36.195801 10274 log.cc:1196] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/5fcdd4cbd7b34f1bb963ac15114a59ab
I20260810 14:31:36.196211 10274 ts_tablet_manager.cc:1939] T 5fcdd4cbd7b34f1bb963ac15114a59ab P f3e9412ec942418d8aa01581235ffbd0: Deleting consensus metadata
I20260810 14:31:36.196319 10275 ts_tablet_manager.cc:1918] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:31:36.196513 10275 log.cc:1196] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/5fcdd4cbd7b34f1bb963ac15114a59ab
I20260810 14:31:36.197017 10275 ts_tablet_manager.cc:1939] T 5fcdd4cbd7b34f1bb963ac15114a59ab P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Deleting consensus metadata
I20260810 14:31:36.197746 10273 ts_tablet_manager.cc:1905] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:36.201956 10289 tablet_bootstrap.cc:492] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Bootstrap starting.
I20260810 14:31:36.202194 9257 catalog_manager.cc:4841] TS 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153): tablet 5fcdd4cbd7b34f1bb963ac15114a59ab (table test-table [id=805ddb79bd004838b2d547227dbc9153]) successfully deleted
I20260810 14:31:36.202461 10273 ts_tablet_manager.cc:1918] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:31:36.202589 10273 log.cc:1196] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/5fcdd4cbd7b34f1bb963ac15114a59ab
I20260810 14:31:36.202837 9257 catalog_manager.cc:4841] TS f3e9412ec942418d8aa01581235ffbd0 (127.6.46.194:42141): tablet 5fcdd4cbd7b34f1bb963ac15114a59ab (table test-table [id=805ddb79bd004838b2d547227dbc9153]) successfully deleted
I20260810 14:31:36.202957 10273 ts_tablet_manager.cc:1939] T 5fcdd4cbd7b34f1bb963ac15114a59ab P bd697cce9db247adb522296cbbf1d939: Deleting consensus metadata
I20260810 14:31:36.203351 9766 tablet_service.cc:1467] Processing CreateTablet for tablet d25b17ece16940e8870521c5d1a31136 (DEFAULT_TABLE table=test-table [id=ac23e3f435bf4ff991f495d7747ee447]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:36.203698 9766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d25b17ece16940e8870521c5d1a31136. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:36.203749 10289 tablet_bootstrap.cc:654] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:36.203971 9259 catalog_manager.cc:4841] TS bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451): tablet 5fcdd4cbd7b34f1bb963ac15114a59ab (table test-table [id=805ddb79bd004838b2d547227dbc9153]) successfully deleted
I20260810 14:31:36.204619 10288 tablet_bootstrap.cc:492] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939: Bootstrap starting.
I20260810 14:31:36.205808 10288 tablet_bootstrap.cc:654] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:36.207553 10288 tablet_bootstrap.cc:492] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939: No bootstrap required, opened a new log
I20260810 14:31:36.207558 10289 tablet_bootstrap.cc:492] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: No bootstrap required, opened a new log
I20260810 14:31:36.207718 10288 ts_tablet_manager.cc:1397] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 14:31:36.207718 10289 ts_tablet_manager.cc:1397] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260810 14:31:36.208072 10293 tablet_bootstrap.cc:492] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0: Bootstrap starting.
I20260810 14:31:36.208397 10288 raft_consensus.cc:348] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:36.208397 10289 raft_consensus.cc:348] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:36.208629 10289 raft_consensus.cc:374] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:36.208626 10288 raft_consensus.cc:374] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:36.208698 10289 raft_consensus.cc:729] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95502ddb7e9f4ab493cf1ef1e3ea7e4a, State: Initialized, Role: FOLLOWER
I20260810 14:31:36.208709 10288 raft_consensus.cc:729] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd697cce9db247adb522296cbbf1d939, State: Initialized, Role: FOLLOWER
I20260810 14:31:36.208925 10288 consensus_queue.cc:260] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:36.208925 10289 consensus_queue.cc:260] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:36.209374 10293 tablet_bootstrap.cc:654] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:36.209432 10288 ts_tablet_manager.cc:1428] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 14:31:36.209438 10289 ts_tablet_manager.cc:1428] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 14:31:36.212045 10293 tablet_bootstrap.cc:492] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0: No bootstrap required, opened a new log
I20260810 14:31:36.212198 10293 ts_tablet_manager.cc:1397] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 14:31:36.212922 10293 raft_consensus.cc:348] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:36.213172 10293 raft_consensus.cc:374] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:36.213378 10293 raft_consensus.cc:729] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e9412ec942418d8aa01581235ffbd0, State: Initialized, Role: FOLLOWER
I20260810 14:31:36.213595 10293 consensus_queue.cc:260] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:36.214260 10293 ts_tablet_manager.cc:1428] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 14:31:36.249980 10249 raft_consensus.cc:482] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:31:36.250201 10249 raft_consensus.cc:504] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:36.250757 10249 leader_election.cc:288] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451), 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153)
I20260810 14:31:36.251276 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25b17ece16940e8870521c5d1a31136" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd697cce9db247adb522296cbbf1d939" is_pre_election: true
I20260810 14:31:36.251309 10043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25b17ece16940e8870521c5d1a31136" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" is_pre_election: true
I20260810 14:31:36.251530 10043 raft_consensus.cc:2454] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 0.
I20260810 14:31:36.251528 9562 raft_consensus.cc:2454] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 0.
I20260810 14:31:36.251935 9681 leader_election.cc:302] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95502ddb7e9f4ab493cf1ef1e3ea7e4a, f3e9412ec942418d8aa01581235ffbd0; no voters:
I20260810 14:31:36.252249 10249 raft_consensus.cc:2781] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:31:36.252346 10249 raft_consensus.cc:482] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:31:36.252454 10249 raft_consensus.cc:3037] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:36.254781 10249 raft_consensus.cc:504] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:36.255317 10249 leader_election.cc:288] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 election: Requested vote from peers bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451), 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153)
I20260810 14:31:36.255736 9562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25b17ece16940e8870521c5d1a31136" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd697cce9db247adb522296cbbf1d939"
I20260810 14:31:36.255769 10043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d25b17ece16940e8870521c5d1a31136" candidate_uuid: "f3e9412ec942418d8aa01581235ffbd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a"
I20260810 14:31:36.255945 10043 raft_consensus.cc:3037] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:36.255962 9562 raft_consensus.cc:3037] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:36.258863 10043 raft_consensus.cc:2454] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 1.
I20260810 14:31:36.258952 9562 raft_consensus.cc:2454] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e9412ec942418d8aa01581235ffbd0 in term 1.
I20260810 14:31:36.259287 9681 leader_election.cc:302] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95502ddb7e9f4ab493cf1ef1e3ea7e4a, f3e9412ec942418d8aa01581235ffbd0; no voters:
I20260810 14:31:36.259536 10249 raft_consensus.cc:2781] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:36.259684 10249 raft_consensus.cc:686] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [term 1 LEADER]: Becoming Leader. State: Replica: f3e9412ec942418d8aa01581235ffbd0, State: Running, Role: LEADER
I20260810 14:31:36.259912 10249 consensus_queue.cc:237] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } }
I20260810 14:31:36.262321 9285 catalog_manager.cc:5495] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3e9412ec942418d8aa01581235ffbd0 (127.6.46.194). New cstate: current_term: 1 leader_uuid: "f3e9412ec942418d8aa01581235ffbd0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3e9412ec942418d8aa01581235ffbd0" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 } health_report { overall_health: UNKNOWN } } }
I20260810 14:31:36.283192 10043 raft_consensus.cc:1261] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Refusing update from remote peer f3e9412ec942418d8aa01581235ffbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:36.283493 9562 raft_consensus.cc:1261] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Refusing update from remote peer f3e9412ec942418d8aa01581235ffbd0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:36.283717 10224 consensus_queue.cc:1035] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95502ddb7e9f4ab493cf1ef1e3ea7e4a" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:36.283975 10249 consensus_queue.cc:1035] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd697cce9db247adb522296cbbf1d939" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 34451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 14:31:36.301195 10242 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 14:31:37.874843 9282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:37.875072 9282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:37.878885 9282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f79ed22c59fd4f099e18bbced4042744: Sending DeleteTablet for 3 replicas of tablet d25b17ece16940e8870521c5d1a31136
I20260810 14:31:37.879565 9516 tablet_service.cc:1514] Processing DeleteTablet for tablet d25b17ece16940e8870521c5d1a31136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:37 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:42724
I20260810 14:31:37.879572 9766 tablet_service.cc:1514] Processing DeleteTablet for tablet d25b17ece16940e8870521c5d1a31136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:37 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:43526
I20260810 14:31:37.879720 10011 tablet_service.cc:1514] Processing DeleteTablet for tablet d25b17ece16940e8870521c5d1a31136 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:37 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:41438
I20260810 14:31:37.880237 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9411
I20260810 14:31:37.882002 10359 tablet_replica.cc:330] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939: stopping tablet replica
I20260810 14:31:37.882216 10359 raft_consensus.cc:2229] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:37.882436 10359 raft_consensus.cc:2258] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:37.883500 10359 ts_tablet_manager.cc:1905] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:37.890935 10359 ts_tablet_manager.cc:1918] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:31:37.891155 10359 log.cc:1196] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/d25b17ece16940e8870521c5d1a31136
I20260810 14:31:37.891474 10361 tablet_replica.cc:330] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0: stopping tablet replica
I20260810 14:31:37.891726 10359 ts_tablet_manager.cc:1939] T d25b17ece16940e8870521c5d1a31136 P bd697cce9db247adb522296cbbf1d939: Deleting consensus metadata
I20260810 14:31:37.891942 10362 tablet_replica.cc:330] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: stopping tablet replica
I20260810 14:31:37.892161 10362 raft_consensus.cc:2229] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:37.892203 10361 raft_consensus.cc:2229] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:31:37.892483 10362 raft_consensus.cc:2258] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:37.892623 10361 raft_consensus.cc:2258] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:37.892812 9259 catalog_manager.cc:4841] TS bd697cce9db247adb522296cbbf1d939 (127.6.46.193:34451): tablet d25b17ece16940e8870521c5d1a31136 (table test-table [id=ac23e3f435bf4ff991f495d7747ee447]) successfully deleted
I20260810 14:31:37.893481 10362 ts_tablet_manager.cc:1905] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:37.894090 10361 ts_tablet_manager.cc:1905] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:37.899880 10362 ts_tablet_manager.cc:1918] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:31:37.900012 10362 log.cc:1196] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/d25b17ece16940e8870521c5d1a31136
I20260810 14:31:37.900424 10361 ts_tablet_manager.cc:1918] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 14:31:37.900496 10362 ts_tablet_manager.cc:1939] T d25b17ece16940e8870521c5d1a31136 P 95502ddb7e9f4ab493cf1ef1e3ea7e4a: Deleting consensus metadata
I20260810 14:31:37.900560 10361 log.cc:1196] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/d25b17ece16940e8870521c5d1a31136
I20260810 14:31:37.901003 10361 ts_tablet_manager.cc:1939] T d25b17ece16940e8870521c5d1a31136 P f3e9412ec942418d8aa01581235ffbd0: Deleting consensus metadata
I20260810 14:31:37.901578 9257 catalog_manager.cc:4841] TS 95502ddb7e9f4ab493cf1ef1e3ea7e4a (127.6.46.195:42153): tablet d25b17ece16940e8870521c5d1a31136 (table test-table [id=ac23e3f435bf4ff991f495d7747ee447]) successfully deleted
I20260810 14:31:37.901966 9257 catalog_manager.cc:4841] TS f3e9412ec942418d8aa01581235ffbd0 (127.6.46.194:42141): tablet d25b17ece16940e8870521c5d1a31136 (table test-table [id=ac23e3f435bf4ff991f495d7747ee447]) successfully deleted
Aug 10 14:31:37 dist-test-slave-q9h9 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:31:37.909307 10365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:37.881598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35212 (local address 127.6.46.193:34451)
0810 14:31:37.890321 (+ 8723us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:37.890327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:37.891352 (+ 1025us) server_negotiation.cc:407] Connection header received
0810 14:31:37.891413 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:37.891418 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:37.891505 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:37.891603 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:37.893544 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:37.894609 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:37.895958 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:37.896350 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:37.896454 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:37.904013 (+ 7559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:37.904028 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:37.904046 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:37.904075 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:37.906792 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:37.907392 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:37.907399 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:37.907401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:37.907556 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:37.907908 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:37.907911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:37.907913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:37.908258 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:37.908525 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:37.908866 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 14:31:37.909032 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8591,"thread_start_us":8487,"threads_started":1}
W20260810 14:31:37.909899 9640 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35212
W20260810 14:31:37.910509 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9411: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:31:37.910703 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 9411
I20260810 14:31:37.936861 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9659
Aug 10 14:31:37 dist-test-slave-q9h9 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:31:37.951116 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:37.937534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.6.46.194:42141)
0810 14:31:37.938002 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:37.938007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:37.938041 (+ 34us) server_negotiation.cc:407] Connection header received
0810 14:31:37.938096 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:37.938100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:37.938167 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:37.938274 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:37.939479 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:37.940219 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:37.941466 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:37.941868 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:37.941964 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:37.945976 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:37.945989 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:37.946004 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:37.946069 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:37.948836 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:37.949400 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:37.949406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:37.949408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:37.949528 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:37.949796 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:37.949799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:37.949801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:37.950227 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:37.950473 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:37.950693 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 14:31:37.950853 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":247,"threads_started":1}
W20260810 14:31:37.951735 9873 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35360
W20260810 14:31:37.952307 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9659: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:31:37.952450 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 9659
I20260810 14:31:37.984663 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9893
Aug 10 14:31:37 dist-test-slave-q9h9 krb5kdc[9213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372295, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:31:37.999111 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:37.985323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53180 (local address 127.6.46.195:42153)
0810 14:31:37.985800 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:37.985805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:37.985821 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:31:37.985872 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:37.985876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:37.985935 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:37.986067 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:37.987453 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:37.988171 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:37.989397 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:37.989899 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:37.990042 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:37.994094 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:37.994109 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:37.994126 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:37.994165 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:37.996716 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:37.997301 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:37.997308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:37.997310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:37.997443 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:37.997788 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:37.997791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:37.997793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:37.998193 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:37.998409 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:37.998637 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 14:31:37.998828 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":229,"threads_started":1}
W20260810 14:31:37.999775 10120 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53180
W20260810 14:31:38.000435 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9893: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:31:38.000577 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 9893
I20260810 14:31:38.024850 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9234
I20260810 14:31:38.025944 9356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:38.140977 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 9234
2026-08-10T14:31:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9466 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:31:38 dist-test-slave-q9h9 krb5kdc[10378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:31:38 dist-test-slave-q9h9 krb5kdc[10378](info): set up 2 sockets
Aug 10 14:31:38 dist-test-slave-q9h9 krb5kdc[10378](info): commencing operation
krb5kdc: starting...
W20260810 14:31:39.348518 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.159s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:31:39 dist-test-slave-q9h9 krb5kdc[10378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:31:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:31:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:31:39.522451 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:40569
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:40569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:39.830811 10401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:39.831073 10401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:39.831208 10401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:39.838179 10401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:31:39.838277 10401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:39.838351 10401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:39.838433 10401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:31:39.838495 10401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:31:39.847776 10401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:40569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:40569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:39.848884 10401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:39.850090 10401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:39.861064 10416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:39.861145 10411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:39.861147 10412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:39.861471 10401 server_base.cc:1034] running on GCE node
I20260810 14:31:39.862361 10401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:39.863622 10401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:39.864832 10401 hybrid_clock.cc:648] HybridClock initialized: now 1786372299864788 us; error 24 us; skew 500 ppm
I20260810 14:31:39.865152 10401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:39 dist-test-slave-q9h9 krb5kdc[10378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372299, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:39.868153 10401 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:31:39.869519 10401 webserver.cc:460] Webserver started at http://127.6.46.254:35057/ using document root <none> and password file <none>
I20260810 14:31:39.870229 10401 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:39.870347 10401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:39.870645 10401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:39.873456 10401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "271fe410b26c40eaa54d311b99c67baf"
format_stamp: "Formatted at 2026-08-10 14:31:39 on dist-test-slave-q9h9"
server_key: "9320d09a45a28fe7e21a71790c506c31"
server_key_iv: "f0a73142f0a9c4f475192c9ee196528e"
server_key_version: "encryptionkey@0"
I20260810 14:31:39.874258 10401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "271fe410b26c40eaa54d311b99c67baf"
format_stamp: "Formatted at 2026-08-10 14:31:39 on dist-test-slave-q9h9"
server_key: "9320d09a45a28fe7e21a71790c506c31"
server_key_iv: "f0a73142f0a9c4f475192c9ee196528e"
server_key_version: "encryptionkey@0"
I20260810 14:31:39.879142 10401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 14:31:39.882297 10429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:39.883436 10401 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:31:39.883581 10401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "271fe410b26c40eaa54d311b99c67baf"
format_stamp: "Formatted at 2026-08-10 14:31:39 on dist-test-slave-q9h9"
server_key: "9320d09a45a28fe7e21a71790c506c31"
server_key_iv: "f0a73142f0a9c4f475192c9ee196528e"
server_key_version: "encryptionkey@0"
I20260810 14:31:39.883750 10401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:39.899679 10401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:39.903015 10401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:39.903247 10401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:39.919502 10401 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:40569
I20260810 14:31:39.919552 10522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:40569 every 8 connection(s)
I20260810 14:31:39.920701 10401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:31:39.924015 10523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:39.926565 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 10401
I20260810 14:31:39.926757 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:31:39.927384 6331 external_mini_cluster.cc:1428] Setting key b90afab06f88a5cdc8305b53267a461b
I20260810 14:31:39.935915 10523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf: Bootstrap starting.
I20260810 14:31:39.939167 10523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:39.940068 10523 log.cc:824] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:31:39 dist-test-slave-q9h9 krb5kdc[10378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:39.942832 10523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf: No bootstrap required, opened a new log
I20260810 14:31:39.948992 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:39.929147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39118 (local address 127.6.46.254:40569)
0810 14:31:39.929917 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:39.929931 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:39.929978 (+ 47us) server_negotiation.cc:407] Connection header received
0810 14:31:39.931947 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:39.931974 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:39.932462 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:39.932870 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:39.934164 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:39.936043 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:39.937254 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:39.938248 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:39.938416 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:39.942251 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:39.942258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:39.942271 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:39.942304 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:39.945210 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:39.945968 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:39.945973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:39.945978 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:39.946175 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:39.946542 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:39.946545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:39.946547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:39.947012 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:39.947241 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:39.947639 (+ 398us) server_negotiation.cc:299] Negotiation successful
0810 14:31:39.947930 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":334,"threads_started":1}
I20260810 14:31:39.950966 10523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271fe410b26c40eaa54d311b99c67baf" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 40569 } }
I20260810 14:31:39.951293 10523 raft_consensus.cc:374] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:39.951399 10523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 271fe410b26c40eaa54d311b99c67baf, State: Initialized, Role: FOLLOWER
I20260810 14:31:39.951915 10523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271fe410b26c40eaa54d311b99c67baf" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 40569 } }
I20260810 14:31:39.952114 10523 raft_consensus.cc:388] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:31:39.952222 10523 raft_consensus.cc:482] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:31:39.952329 10523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:39.954245 10523 raft_consensus.cc:504] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271fe410b26c40eaa54d311b99c67baf" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 40569 } }
I20260810 14:31:39.954641 10523 leader_election.cc:302] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 271fe410b26c40eaa54d311b99c67baf; no voters:
I20260810 14:31:39.955195 10523 leader_election.cc:288] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:31:39.955292 10534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:39.956151 10534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [term 1 LEADER]: Becoming Leader. State: Replica: 271fe410b26c40eaa54d311b99c67baf, State: Running, Role: LEADER
I20260810 14:31:39.956589 10534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271fe410b26c40eaa54d311b99c67baf" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 40569 } }
I20260810 14:31:39.956991 10523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:31:39.959033 10537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 271fe410b26c40eaa54d311b99c67baf. Latest consensus state: current_term: 1 leader_uuid: "271fe410b26c40eaa54d311b99c67baf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271fe410b26c40eaa54d311b99c67baf" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 40569 } } }
I20260810 14:31:39.959372 10537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:39.959914 10535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "271fe410b26c40eaa54d311b99c67baf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "271fe410b26c40eaa54d311b99c67baf" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 40569 } } }
I20260810 14:31:39.960139 10535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:39.960240 10547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:31:39.967875 10547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:31:39.975590 10547 catalog_manager.cc:1347] Generated new cluster ID: a496576b7e87490795e2e79cb2fc30e3
I20260810 14:31:39.975759 10547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:31:39.985311 10547 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:31:39.986716 10547 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:31:40.000252 10547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 271fe410b26c40eaa54d311b99c67baf: Generated new TSK 0
I20260810 14:31:40.000895 10547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:31:40.057078 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:40569
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:40.391937 10569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:40.392189 10569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:40.392328 10569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:40.399286 10569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:40.399377 10569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:40.399521 10569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:31:40.408672 10569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:40569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:40.409777 10569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:40.410987 10569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:40.423206 10576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:40.423218 10579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:40.423170 10583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:40.424077 10569 server_base.cc:1034] running on GCE node
I20260810 14:31:40.424620 10569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:40.425427 10569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:40.426617 10569 hybrid_clock.cc:648] HybridClock initialized: now 1786372300426564 us; error 45 us; skew 500 ppm
I20260810 14:31:40.426890 10569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:40 dist-test-slave-q9h9 krb5kdc[10378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372300, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:40.429756 10569 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:31:40.431103 10569 webserver.cc:460] Webserver started at http://127.6.46.193:38865/ using document root <none> and password file <none>
I20260810 14:31:40.431728 10569 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:40.431841 10569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:40.432112 10569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:40.434854 10569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "08b1e2256b7e4f6b9d7a38b3af396175"
format_stamp: "Formatted at 2026-08-10 14:31:40 on dist-test-slave-q9h9"
server_key: "104fa5c0c94258ed77f16ad09ee15bcb"
server_key_iv: "a93c141ceb026c0396a3da40a792702c"
server_key_version: "encryptionkey@0"
I20260810 14:31:40.435590 10569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "08b1e2256b7e4f6b9d7a38b3af396175"
format_stamp: "Formatted at 2026-08-10 14:31:40 on dist-test-slave-q9h9"
server_key: "104fa5c0c94258ed77f16ad09ee15bcb"
server_key_iv: "a93c141ceb026c0396a3da40a792702c"
server_key_version: "encryptionkey@0"
I20260810 14:31:40.440346 10569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:31:40.443536 10595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:40.444715 10569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:31:40.444861 10569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "08b1e2256b7e4f6b9d7a38b3af396175"
format_stamp: "Formatted at 2026-08-10 14:31:40 on dist-test-slave-q9h9"
server_key: "104fa5c0c94258ed77f16ad09ee15bcb"
server_key_iv: "a93c141ceb026c0396a3da40a792702c"
server_key_version: "encryptionkey@0"
I20260810 14:31:40.445030 10569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:40.457319 10569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:40.460970 10569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:40.461222 10569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:40.462127 10569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:40.463423 10569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:40.463501 10569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:40.463625 10569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:40.463676 10569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:40.493461 10569 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:41865
I20260810 14:31:40.493503 10785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:41865 every 8 connection(s)
I20260810 14:31:40.494683 10569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:31:40.495199 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 10569
I20260810 14:31:40.495371 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:31:40.496220 6331 external_mini_cluster.cc:1428] Setting key 3a658feae36872c75ddb40fab4cb71e1
Aug 10 14:31:40 dist-test-slave-q9h9 krb5kdc[10378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372300, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:40.514568 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:40.497099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:53103 (local address 127.6.46.254:40569)
0810 14:31:40.497509 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:40.497515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:40.498528 (+ 1013us) server_negotiation.cc:407] Connection header received
0810 14:31:40.500434 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:40.500440 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:40.500524 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:40.500621 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:40.502100 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:40.503052 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:40.504495 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:40.505018 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:40.505138 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:40.509093 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:40.509098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:40.509101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:40.509133 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:40.511514 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:40.512333 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:40.512337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:40.512339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:40.512445 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:40.512903 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:40.512906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:40.512908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:40.513138 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:40.513245 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:40.514167 (+ 922us) server_negotiation.cc:299] Negotiation successful
0810 14:31:40.514324 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":199,"threads_started":1}
I20260810 14:31:40.515991 10790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:40.497546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:40569 (local address 127.6.46.193:53103)
0810 14:31:40.498347 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:40.498405 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:40.500165 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:40.500847 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:40.500855 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:40.501328 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:40.501874 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:40.501899 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:40.503190 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:40.503194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:40.504356 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:40.504368 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:40.505259 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:40.505266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:40.505489 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:40.506150 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:40.506184 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:40.508876 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:40.511669 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:40.511678 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:40.511692 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:40.512229 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:40.512545 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:40.512551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:40.512555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:40.512805 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:40.513273 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:40.513282 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:40.513728 (+ 446us) client_negotiation.cc:769] Sending connection context
0810 14:31:40.514087 (+ 359us) client_negotiation.cc:240] Negotiation successful
0810 14:31:40.514610 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":292,"threads_started":1}
I20260810 14:31:40.517486 10786 heartbeater.cc:347] Connected to a master server at 127.6.46.254:40569
I20260810 14:31:40.517833 10786 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
I20260810 14:31:40.518771 10786 heartbeater.cc:511] Master 127.6.46.254:40569 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
I20260810 14:31:40.520768 10454 ts_manager.cc:194] Registered new tserver with Master: 08b1e2256b7e4f6b9d7a38b3af396175 (127.6.46.193:41865)
I20260810 14:31:40.522673 10454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:53103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:31:40.541863 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:40569
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:40.849682 10794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:40.849939 10794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:40.850093 10794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:40.857028 10794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:40.857133 10794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:40.857297 10794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:31:40.866420 10794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:40569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:40.867549 10794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:40.868736 10794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:40.881014 10808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:40.881057 10805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:40.881086 10812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:40.881891 10794 server_base.cc:1034] running on GCE node
I20260810 14:31:40.882445 10794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:40.883283 10794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:40.884478 10794 hybrid_clock.cc:648] HybridClock initialized: now 1786372300884429 us; error 40 us; skew 500 ppm
I20260810 14:31:40.884761 10794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:40 dist-test-slave-q9h9 krb5kdc[10378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372300, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:40.887753 10794 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:31:40.889048 10794 webserver.cc:460] Webserver started at http://127.6.46.194:35321/ using document root <none> and password file <none>
I20260810 14:31:40.889659 10794 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:40.889775 10794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:40.890098 10794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:40.892778 10794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "79cd010831d043d8b1d37ae15b9d79ce"
format_stamp: "Formatted at 2026-08-10 14:31:40 on dist-test-slave-q9h9"
server_key: "76f2bfd0e4f129cd4acca4d59f3ba887"
server_key_iv: "9af814ff8cf4e0ce2ee509e1918ee4e5"
server_key_version: "encryptionkey@0"
I20260810 14:31:40.893508 10794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "79cd010831d043d8b1d37ae15b9d79ce"
format_stamp: "Formatted at 2026-08-10 14:31:40 on dist-test-slave-q9h9"
server_key: "76f2bfd0e4f129cd4acca4d59f3ba887"
server_key_iv: "9af814ff8cf4e0ce2ee509e1918ee4e5"
server_key_version: "encryptionkey@0"
I20260810 14:31:40.898236 10794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260810 14:31:40.901443 10830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:40.902667 10794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:31:40.902815 10794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "79cd010831d043d8b1d37ae15b9d79ce"
format_stamp: "Formatted at 2026-08-10 14:31:40 on dist-test-slave-q9h9"
server_key: "76f2bfd0e4f129cd4acca4d59f3ba887"
server_key_iv: "9af814ff8cf4e0ce2ee509e1918ee4e5"
server_key_version: "encryptionkey@0"
I20260810 14:31:40.902990 10794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:40.931412 10794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:40.935066 10794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:40.935321 10794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:40.936228 10794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:40.937515 10794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:40.937593 10794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:40.937718 10794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:40.937781 10794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:40.968809 10794 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:38833
I20260810 14:31:40.968825 11012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:38833 every 8 connection(s)
I20260810 14:31:40.970008 10794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:31:40.976934 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 10794
I20260810 14:31:40.977118 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:31:40.977707 6331 external_mini_cluster.cc:1428] Setting key 5cd895facedb03e760e68effb51182ad
Aug 10 14:31:40 dist-test-slave-q9h9 krb5kdc[10378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372300, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:40.989425 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:40.972375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:60881 (local address 127.6.46.254:40569)
0810 14:31:40.972535 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:40.972540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:40.973605 (+ 1065us) server_negotiation.cc:407] Connection header received
0810 14:31:40.975425 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:40.975431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:40.975507 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:40.975593 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:40.977349 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:40.978405 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:40.980228 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:40.980642 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:40.980754 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:40.983996 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:40.984002 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:40.984005 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:40.984035 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:40.986492 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:40.987206 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:40.987211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:40.987213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:40.987340 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:40.987772 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:40.987777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:40.987780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:40.988096 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:40.988244 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:40.988865 (+ 621us) server_negotiation.cc:299] Negotiation successful
0810 14:31:40.989088 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:31:40.990304 11020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:40.972788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:40569 (local address 127.6.46.194:60881)
0810 14:31:40.973448 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:40.973489 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:40.975160 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:40.975831 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:40.975838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:40.976308 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:40.977089 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:40.977119 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:40.978562 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:40.978568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:40.980074 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:40.980088 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:40.980854 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:40.980862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:40.981086 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:40.981700 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:40.981720 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:40.983791 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:40.986633 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:40.986642 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:40.986656 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:40.987093 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:40.987443 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:40.987448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:40.987452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:40.987671 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:40.988225 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:40.988232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:40.988533 (+ 301us) client_negotiation.cc:769] Sending connection context
0810 14:31:40.988764 (+ 231us) client_negotiation.cc:240] Negotiation successful
0810 14:31:40.989224 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":275,"threads_started":1}
I20260810 14:31:40.991603 11013 heartbeater.cc:347] Connected to a master server at 127.6.46.254:40569
I20260810 14:31:40.991956 11013 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:40.992733 11013 heartbeater.cc:511] Master 127.6.46.254:40569 requested a full tablet report, sending...
I20260810 14:31:40.994221 10454 ts_manager.cc:194] Registered new tserver with Master: 79cd010831d043d8b1d37ae15b9d79ce (127.6.46.194:38833)
I20260810 14:31:40.994985 10454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:60881
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:31:41.020416 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:40569
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:41.330564 11024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:41.330818 11024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:41.330947 11024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:41.338234 11024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:41.338340 11024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:41.338502 11024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:31:41.347865 11024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:40569
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:41.349148 11024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:41.350451 11024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:41.363181 11030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:41.363215 11031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:41.363111 11037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:41.367364 11024 server_base.cc:1034] running on GCE node
I20260810 14:31:41.367890 11024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:41.368763 11024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:41.369948 11024 hybrid_clock.cc:648] HybridClock initialized: now 1786372301369897 us; error 47 us; skew 500 ppm
I20260810 14:31:41.370271 11024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:41 dist-test-slave-q9h9 krb5kdc[10378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372301, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:41.373189 11024 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:31:41.374732 11024 webserver.cc:460] Webserver started at http://127.6.46.195:42295/ using document root <none> and password file <none>
I20260810 14:31:41.375415 11024 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:41.375531 11024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:41.375823 11024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:41.378669 11024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "941fe42d418b492da8922b6df08f96f1"
format_stamp: "Formatted at 2026-08-10 14:31:41 on dist-test-slave-q9h9"
server_key: "a216e268a897d6fbbd9cd6eb1069c468"
server_key_iv: "dde38063cc3da79b841ab9aa1096a1bc"
server_key_version: "encryptionkey@0"
I20260810 14:31:41.379411 11024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "941fe42d418b492da8922b6df08f96f1"
format_stamp: "Formatted at 2026-08-10 14:31:41 on dist-test-slave-q9h9"
server_key: "a216e268a897d6fbbd9cd6eb1069c468"
server_key_iv: "dde38063cc3da79b841ab9aa1096a1bc"
server_key_version: "encryptionkey@0"
I20260810 14:31:41.384460 11024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 14:31:41.387871 11048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:41.389189 11024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:31:41.389330 11024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "941fe42d418b492da8922b6df08f96f1"
format_stamp: "Formatted at 2026-08-10 14:31:41 on dist-test-slave-q9h9"
server_key: "a216e268a897d6fbbd9cd6eb1069c468"
server_key_iv: "dde38063cc3da79b841ab9aa1096a1bc"
server_key_version: "encryptionkey@0"
I20260810 14:31:41.389518 11024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:41.414606 11024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:41.418318 11024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:41.418586 11024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:41.419513 11024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:41.420747 11024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:41.420818 11024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:41.420928 11024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:41.420974 11024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:41.451581 11024 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:35607
I20260810 14:31:41.451647 11252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:35607 every 8 connection(s)
I20260810 14:31:41.452809 11024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:31:41.455386 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 11024
I20260810 14:31:41.455578 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:31:41.456032 6331 external_mini_cluster.cc:1428] Setting key 883cc84282bdfcd197b6fcc13a43ee42
Aug 10 14:31:41 dist-test-slave-q9h9 krb5kdc[10378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372301, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:41.471972 10788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:41.455467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:42469 (local address 127.6.46.254:40569)
0810 14:31:41.455676 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:41.455683 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:41.457076 (+ 1393us) server_negotiation.cc:407] Connection header received
0810 14:31:41.458960 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:41.458964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:41.459036 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:41.459112 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:41.460489 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:41.461226 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:41.462834 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:41.463271 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:41.463389 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:41.466574 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:41.466580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:41.466583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:41.466615 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:41.469219 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:41.469908 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:41.469913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:41.469916 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:41.470120 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:41.470552 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:41.470557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:41.470559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:41.470860 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:41.470990 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:41.471574 (+ 584us) server_negotiation.cc:299] Negotiation successful
0810 14:31:41.471738 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 14:31:41.472901 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:41.455943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:40569 (local address 127.6.46.195:42469)
0810 14:31:41.456763 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:41.456807 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:41.458700 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:41.459295 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:41.459303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:41.459789 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:41.460315 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:41.460334 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:41.461351 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:41.461354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:41.462657 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:41.462674 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:41.463464 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:41.463469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:41.463685 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:41.464271 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:41.464292 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:41.466377 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:41.469347 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:41.469355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:41.469368 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:41.469785 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:41.470248 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:41.470253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:41.470256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:41.470446 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:41.470975 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:41.470981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:41.471256 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 14:31:41.471479 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 14:31:41.471857 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":353,"threads_started":1}
I20260810 14:31:41.474187 11253 heartbeater.cc:347] Connected to a master server at 127.6.46.254:40569
I20260810 14:31:41.474442 11253 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:41.475103 11253 heartbeater.cc:511] Master 127.6.46.254:40569 requested a full tablet report, sending...
I20260810 14:31:41.476373 10454 ts_manager.cc:194] Registered new tserver with Master: 941fe42d418b492da8922b6df08f96f1 (127.6.46.195:35607)
I20260810 14:31:41.477129 10454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:42469
I20260810 14:31:41.482887 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 14:31:41.516053 10788 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:31:41.513768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.6.46.254:40569)
0810 14:31:41.513934 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:41.513939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:41.514448 (+ 509us) server_negotiation.cc:407] Connection header received
0810 14:31:41.514615 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:41.514619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:41.514678 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:41.514768 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:41.515847 (+ 1079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39124: BlockingRecv error: recv got EOF from 127.0.0.1:39124 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:31:41.519326 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10569
W20260810 14:31:41.521817 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10569: Not authorized: Client connection negotiation failed: client connection to 127.6.46.193:41865: server requires authentication, but client does not have Kerberos credentials available
I20260810 14:31:41.521996 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 10569
I20260810 14:31:41.540920 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10794
W20260810 14:31:41.543220 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10794: Not authorized: Client connection negotiation failed: client connection to 127.6.46.194:38833: server requires authentication, but client does not have Kerberos credentials available
I20260810 14:31:41.543327 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 10794
I20260810 14:31:41.565294 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11024
W20260810 14:31:41.568212 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11024: Not authorized: Client connection negotiation failed: client connection to 127.6.46.195:35607: server requires authentication, but client does not have Kerberos credentials available
I20260810 14:31:41.568367 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 11024
I20260810 14:31:41.589497 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10401
I20260810 14:31:41.590603 10514 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:41.696710 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 10401
2026-08-10T14:31:41Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3554 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:31:41 dist-test-slave-q9h9 krb5kdc[11279](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:31:41 dist-test-slave-q9h9 krb5kdc[11279](info): set up 2 sockets
Aug 10 14:31:41 dist-test-slave-q9h9 krb5kdc[11279](info): commencing operation
krb5kdc: starting...
W20260810 14:31:42.901975 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:31:43 dist-test-slave-q9h9 krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:31:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:31:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:31:43.076421 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36773
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:32777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:36773
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:43.385622 11301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:43.385879 11301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:43.386005 11301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:43.392886 11301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:31:43.392982 11301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:43.393050 11301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:43.393139 11301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:31:43.393200 11301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:31:43.402501 11301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:32777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:36773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:43.403656 11301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:43.404860 11301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:43.415781 11310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:43.415868 11307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:43.415889 11308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:43.416482 11301 server_base.cc:1034] running on GCE node
I20260810 14:31:43.417160 11301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:43.418424 11301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:43.419633 11301 hybrid_clock.cc:648] HybridClock initialized: now 1786372303419577 us; error 26 us; skew 500 ppm
I20260810 14:31:43.419945 11301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:43 dist-test-slave-q9h9 krb5kdc[11279](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372303, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:43.425879 11301 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:31:43.427191 11301 webserver.cc:460] Webserver started at http://127.6.46.254:38277/ using document root <none> and password file <none>
I20260810 14:31:43.427809 11301 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:43.427918 11301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:43.428201 11301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:43.431085 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "2fb7bbc8da2f4ed39c7636b9ef05f6d1"
format_stamp: "Formatted at 2026-08-10 14:31:43 on dist-test-slave-q9h9"
server_key: "6028abaadd078e6d07f0ad65554cbf70"
server_key_iv: "cb7ea36c67718577192e9a0918b0ed51"
server_key_version: "encryptionkey@0"
I20260810 14:31:43.431833 11301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "2fb7bbc8da2f4ed39c7636b9ef05f6d1"
format_stamp: "Formatted at 2026-08-10 14:31:43 on dist-test-slave-q9h9"
server_key: "6028abaadd078e6d07f0ad65554cbf70"
server_key_iv: "cb7ea36c67718577192e9a0918b0ed51"
server_key_version: "encryptionkey@0"
I20260810 14:31:43.436632 11301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 14:31:43.439822 11329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:43.441097 11301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:31:43.441253 11301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "2fb7bbc8da2f4ed39c7636b9ef05f6d1"
format_stamp: "Formatted at 2026-08-10 14:31:43 on dist-test-slave-q9h9"
server_key: "6028abaadd078e6d07f0ad65554cbf70"
server_key_iv: "cb7ea36c67718577192e9a0918b0ed51"
server_key_version: "encryptionkey@0"
I20260810 14:31:43.441432 11301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:43.455525 11301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:43.459319 11301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:43.459568 11301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:43.475891 11301 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:36773
I20260810 14:31:43.475936 11424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:36773 every 8 connection(s)
I20260810 14:31:43.477147 11301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:31:43.480638 11425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:43.480847 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 11301
I20260810 14:31:43.481081 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:31:43.481693 6331 external_mini_cluster.cc:1428] Setting key 4a028180f72da4472dda874f7f66955a
I20260810 14:31:43.493609 11425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1: Bootstrap starting.
Aug 10 14:31:43 dist-test-slave-q9h9 krb5kdc[11279](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:43.497020 11425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:43.498080 11425 log.cc:824] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:43.500653 11425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1: No bootstrap required, opened a new log
I20260810 14:31:43.503679 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:43.483439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41790 (local address 127.6.46.254:36773)
0810 14:31:43.484183 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:43.484193 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:43.484235 (+ 42us) server_negotiation.cc:407] Connection header received
0810 14:31:43.486269 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:43.486291 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:43.486673 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:43.486992 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:43.488744 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:43.489998 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:43.491719 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:43.492331 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:43.492446 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:43.496878 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:43.496887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:43.496899 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:43.496943 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:43.499830 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:43.500456 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:43.500463 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:43.500468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:43.500633 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:43.501036 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:43.501040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:43.501042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:43.501546 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:43.501839 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:43.502343 (+ 504us) server_negotiation.cc:299] Negotiation successful
0810 14:31:43.502717 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":346,"threads_started":1}
I20260810 14:31:43.507452 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:31:43.508791 11425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb7bbc8da2f4ed39c7636b9ef05f6d1" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36773 } }
I20260810 14:31:43.509104 11425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:43.509193 11425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb7bbc8da2f4ed39c7636b9ef05f6d1, State: Initialized, Role: FOLLOWER
I20260810 14:31:43.509694 11425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb7bbc8da2f4ed39c7636b9ef05f6d1" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36773 } }
I20260810 14:31:43.509897 11425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:31:43.510033 11425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:31:43.510198 11425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:43.512187 11425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb7bbc8da2f4ed39c7636b9ef05f6d1" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36773 } }
I20260810 14:31:43.512624 11425 leader_election.cc:302] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fb7bbc8da2f4ed39c7636b9ef05f6d1; no voters:
I20260810 14:31:43.513200 11425 leader_election.cc:288] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:31:43.513296 11430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:43.515352 11425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:31:43.515686 11430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [term 1 LEADER]: Becoming Leader. State: Replica: 2fb7bbc8da2f4ed39c7636b9ef05f6d1, State: Running, Role: LEADER
I20260810 14:31:43.516240 11430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb7bbc8da2f4ed39c7636b9ef05f6d1" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36773 } }
I20260810 14:31:43.519148 11431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fb7bbc8da2f4ed39c7636b9ef05f6d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb7bbc8da2f4ed39c7636b9ef05f6d1" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36773 } } }
I20260810 14:31:43.519891 11432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fb7bbc8da2f4ed39c7636b9ef05f6d1. Latest consensus state: current_term: 1 leader_uuid: "2fb7bbc8da2f4ed39c7636b9ef05f6d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fb7bbc8da2f4ed39c7636b9ef05f6d1" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36773 } } }
I20260810 14:31:43.520170 11432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:43.519479 11431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:43.531371 11451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:31:43.536176 11451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:31:43.547528 11451 catalog_manager.cc:1347] Generated new cluster ID: 25524f046c084908afdc2c7aaaea9583
I20260810 14:31:43.547703 11451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:31:43.559609 11451 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:31:43.561299 11451 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:31:43.563611 11428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:43.555948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.6.46.254:36773)
0810 14:31:43.556122 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:43.556136 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:43.556691 (+ 555us) server_negotiation.cc:407] Connection header received
0810 14:31:43.556893 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:43.556899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:43.556980 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:43.557088 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:43.558228 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:43.559252 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:43.560609 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:43.561018 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:43.561121 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:43.562171 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:43.562176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:43.562180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:43.562831 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:43.562890 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:43.562920 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:43.563011 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:43.563172 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 14:31:43.563297 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:31:43.577589 11451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fb7bbc8da2f4ed39c7636b9ef05f6d1: Generated new TSK 0
I20260810 14:31:43.578307 11451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:31:43.586812 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11301
I20260810 14:31:43.587819 11418 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:43.685595 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 11301
2026-08-10T14:31:43Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1981 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:31:43 dist-test-slave-q9h9 krb5kdc[11477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:31:43 dist-test-slave-q9h9 krb5kdc[11477](info): set up 2 sockets
Aug 10 14:31:43 dist-test-slave-q9h9 krb5kdc[11477](info): commencing operation
krb5kdc: starting...
W20260810 14:31:44.880781 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.155s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:31:44 dist-test-slave-q9h9 krb5kdc[11477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:31:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:31:45.059399 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:38563
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:38563
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:45.366786 11498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:45.367054 11498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:45.367193 11498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:45.374120 11498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:31:45.374209 11498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:45.374277 11498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:45.374353 11498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:31:45.374413 11498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:31:45.383699 11498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:38563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:38563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:45.384889 11498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:45.386140 11498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:45.396921 11511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:45.396986 11512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:45.396955 11515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:45.398307 11498 server_base.cc:1034] running on GCE node
I20260810 14:31:45.399041 11498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:45.400292 11498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:45.401489 11498 hybrid_clock.cc:648] HybridClock initialized: now 1786372305401439 us; error 34 us; skew 500 ppm
I20260810 14:31:45.401803 11498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:45 dist-test-slave-q9h9 krb5kdc[11477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372305, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:45.404768 11498 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:31:45.406108 11498 webserver.cc:460] Webserver started at http://127.6.46.254:45939/ using document root <none> and password file <none>
I20260810 14:31:45.406738 11498 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:45.406848 11498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:45.407125 11498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:45.409897 11498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "3672ede818bf434cb912945d109a9422"
format_stamp: "Formatted at 2026-08-10 14:31:45 on dist-test-slave-q9h9"
server_key: "0a8159a4ba419ba7ad270bf034db50fb"
server_key_iv: "edb1b4517960fd290a1d87d284911b62"
server_key_version: "encryptionkey@0"
I20260810 14:31:45.410653 11498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "3672ede818bf434cb912945d109a9422"
format_stamp: "Formatted at 2026-08-10 14:31:45 on dist-test-slave-q9h9"
server_key: "0a8159a4ba419ba7ad270bf034db50fb"
server_key_iv: "edb1b4517960fd290a1d87d284911b62"
server_key_version: "encryptionkey@0"
I20260810 14:31:45.415432 11498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 14:31:45.418540 11528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:45.419740 11498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:31:45.419888 11498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "3672ede818bf434cb912945d109a9422"
format_stamp: "Formatted at 2026-08-10 14:31:45 on dist-test-slave-q9h9"
server_key: "0a8159a4ba419ba7ad270bf034db50fb"
server_key_iv: "edb1b4517960fd290a1d87d284911b62"
server_key_version: "encryptionkey@0"
I20260810 14:31:45.420058 11498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:45.432817 11498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:45.436621 11498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:45.436863 11498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:45.453504 11498 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:38563
I20260810 14:31:45.453544 11621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:38563 every 8 connection(s)
I20260810 14:31:45.454753 11498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:31:45.458233 11622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:45.463477 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 11498
I20260810 14:31:45.463728 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:31:45.464533 6331 external_mini_cluster.cc:1428] Setting key 20ab738e906bb18d870d21da1ef17ad1
I20260810 14:31:45.468825 11622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422: Bootstrap starting.
I20260810 14:31:45.472373 11622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:45.473546 11622 log.cc:824] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:45.476279 11622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422: No bootstrap required, opened a new log
Aug 10 14:31:45 dist-test-slave-q9h9 krb5kdc[11477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:45.483839 11622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3672ede818bf434cb912945d109a9422" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38563 } }
I20260810 14:31:45.484165 11622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:45.484273 11622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3672ede818bf434cb912945d109a9422, State: Initialized, Role: FOLLOWER
I20260810 14:31:45.484877 11622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3672ede818bf434cb912945d109a9422" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38563 } }
I20260810 14:31:45.485098 11622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:31:45.485237 11622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:31:45.485421 11622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:45.485929 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:45.466564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.6.46.254:38563)
0810 14:31:45.467311 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:45.467321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:45.467358 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:31:45.468952 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:45.468975 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:45.469319 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:45.469600 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:45.471329 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:45.472585 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:45.474393 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:45.475056 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:45.475214 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:45.479135 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:45.479145 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:45.479157 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:45.479201 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:45.482148 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:45.482794 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:45.482800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:45.482805 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:45.482967 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:45.483324 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:45.483328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:45.483330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:45.483817 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:45.484108 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:45.484544 (+ 436us) server_negotiation.cc:299] Negotiation successful
0810 14:31:45.484896 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":355,"threads_started":1}
I20260810 14:31:45.488052 11622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3672ede818bf434cb912945d109a9422" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38563 } }
I20260810 14:31:45.488548 11622 leader_election.cc:302] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3672ede818bf434cb912945d109a9422; no voters:
I20260810 14:31:45.489126 11622 leader_election.cc:288] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:31:45.489215 11629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:45.490182 11629 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [term 1 LEADER]: Becoming Leader. State: Replica: 3672ede818bf434cb912945d109a9422, State: Running, Role: LEADER
I20260810 14:31:45.490653 11629 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3672ede818bf434cb912945d109a9422" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38563 } }
I20260810 14:31:45.491096 11622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:31:45.493155 11631 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3672ede818bf434cb912945d109a9422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3672ede818bf434cb912945d109a9422" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38563 } } }
I20260810 14:31:45.493571 11631 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:45.493462 11634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3672ede818bf434cb912945d109a9422. Latest consensus state: current_term: 1 leader_uuid: "3672ede818bf434cb912945d109a9422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3672ede818bf434cb912945d109a9422" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38563 } } }
I20260810 14:31:45.493763 11634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:45.495047 11642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:31:45.500131 11642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:31:45.507987 11642 catalog_manager.cc:1347] Generated new cluster ID: 3d2c5eb299ed407ab0ea29c9ceb1c5b4
I20260810 14:31:45.508090 11642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:31:45.523739 11642 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:31:45.525074 11642 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:31:45.530550 11642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3672ede818bf434cb912945d109a9422: Generated new TSK 0
I20260810 14:31:45.531226 11642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:31:45.579741 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38563
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:45.903395 11663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:45.903636 11663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:45.903750 11663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:45.910658 11663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:45.910754 11663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:45.910912 11663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:31:45.920116 11663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:45.921223 11663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:45.922472 11663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:45.934661 11674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:45.934736 11669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:45.934770 11671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:45.935338 11663 server_base.cc:1034] running on GCE node
I20260810 14:31:45.935861 11663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:45.936733 11663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:45.937937 11663 hybrid_clock.cc:648] HybridClock initialized: now 1786372305937890 us; error 35 us; skew 500 ppm
I20260810 14:31:45.938251 11663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:45 dist-test-slave-q9h9 krb5kdc[11477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372305, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:45.941229 11663 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:31:45.942548 11663 webserver.cc:460] Webserver started at http://127.6.46.193:39229/ using document root <none> and password file <none>
I20260810 14:31:45.943187 11663 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:45.943303 11663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:45.943575 11663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:45.946358 11663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "4c0a4f35c2f948cd8382edbce3ceac6a"
format_stamp: "Formatted at 2026-08-10 14:31:45 on dist-test-slave-q9h9"
server_key: "332ff81f0ec6f9ab3589e8e453d96b1b"
server_key_iv: "21e979e7665184ae8aaff342a0ab550f"
server_key_version: "encryptionkey@0"
I20260810 14:31:45.947099 11663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "4c0a4f35c2f948cd8382edbce3ceac6a"
format_stamp: "Formatted at 2026-08-10 14:31:45 on dist-test-slave-q9h9"
server_key: "332ff81f0ec6f9ab3589e8e453d96b1b"
server_key_iv: "21e979e7665184ae8aaff342a0ab550f"
server_key_version: "encryptionkey@0"
I20260810 14:31:45.951876 11663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:31:45.955065 11688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:45.956221 11663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:31:45.956362 11663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "4c0a4f35c2f948cd8382edbce3ceac6a"
format_stamp: "Formatted at 2026-08-10 14:31:45 on dist-test-slave-q9h9"
server_key: "332ff81f0ec6f9ab3589e8e453d96b1b"
server_key_iv: "21e979e7665184ae8aaff342a0ab550f"
server_key_version: "encryptionkey@0"
I20260810 14:31:45.956530 11663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:45.977631 11663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:45.981451 11663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:45.981709 11663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:45.982599 11663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:45.983857 11663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:45.983933 11663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:45.984053 11663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:45.984107 11663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:46.013536 11663 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:44985
I20260810 14:31:46.013628 11864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:44985 every 8 connection(s)
I20260810 14:31:46.014765 11663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:31:46.024078 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 11663
I20260810 14:31:46.024276 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:31:46.024968 6331 external_mini_cluster.cc:1428] Setting key 1905d23524ecd3811fa3c2ce79f34131
Aug 10 14:31:46 dist-test-slave-q9h9 krb5kdc[11477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372305, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:46.034220 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:46.017119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:54709 (local address 127.6.46.254:38563)
0810 14:31:46.017514 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:46.017519 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:46.018373 (+ 854us) server_negotiation.cc:407] Connection header received
0810 14:31:46.020117 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:46.020123 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:46.020201 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:46.020301 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:46.021713 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:46.022477 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:46.024122 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:46.024647 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:46.024765 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:46.028517 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:46.028522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:46.028525 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:46.028557 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:46.031294 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:46.032072 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:46.032076 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:46.032078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:46.032205 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:46.032646 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:46.032649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:46.032651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:46.032912 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:46.033021 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:46.033751 (+ 730us) server_negotiation.cc:299] Negotiation successful
0810 14:31:46.033907 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":203,"threads_started":1}
I20260810 14:31:46.035230 11871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:46.017515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:38563 (local address 127.6.46.193:54709)
0810 14:31:46.018228 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:46.018265 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:46.019853 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:46.020531 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:46.020547 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:46.021009 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:46.021534 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:46.021562 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:46.022628 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:46.022632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:46.023984 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:46.023997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:46.024861 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:46.024868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:46.025107 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:46.025722 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:46.025751 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:46.028311 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:46.031421 (+ 3110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:46.031430 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:46.031444 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:46.031968 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:46.032312 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:46.032316 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:46.032319 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:46.032544 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:46.033031 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:46.033037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:46.033398 (+ 361us) client_negotiation.cc:769] Sending connection context
0810 14:31:46.033667 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 14:31:46.034161 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":264,"threads_started":1}
I20260810 14:31:46.036559 11866 heartbeater.cc:347] Connected to a master server at 127.6.46.254:38563
I20260810 14:31:46.036837 11866 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:46.037482 11866 heartbeater.cc:511] Master 127.6.46.254:38563 requested a full tablet report, sending...
I20260810 14:31:46.039386 11563 ts_manager.cc:194] Registered new tserver with Master: 4c0a4f35c2f948cd8382edbce3ceac6a (127.6.46.193:44985)
I20260810 14:31:46.040807 11563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:54709
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:31:46.066823 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38563
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:46.381309 11875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:46.381572 11875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:46.381716 11875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:46.388738 11875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:46.388841 11875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:46.389019 11875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:31:46.398536 11875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:46.399720 11875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:46.401001 11875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:46.413348 11885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:46.413394 11888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:46.413419 11884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:46.414361 11875 server_base.cc:1034] running on GCE node
I20260810 14:31:46.414876 11875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:46.415692 11875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:46.416888 11875 hybrid_clock.cc:648] HybridClock initialized: now 1786372306416844 us; error 31 us; skew 500 ppm
I20260810 14:31:46.417191 11875 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:46 dist-test-slave-q9h9 krb5kdc[11477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372306, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:46.420161 11875 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:31:46.421434 11875 webserver.cc:460] Webserver started at http://127.6.46.194:46573/ using document root <none> and password file <none>
I20260810 14:31:46.422077 11875 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:46.422192 11875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:46.422475 11875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:46.425334 11875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "8c06c832b9d14c9cb00e58190b2164a1"
format_stamp: "Formatted at 2026-08-10 14:31:46 on dist-test-slave-q9h9"
server_key: "76f3778562eed4119e82266b4f56b025"
server_key_iv: "5a6ae8c4aa9bfcf086d8eb7cd7aee913"
server_key_version: "encryptionkey@0"
I20260810 14:31:46.426102 11875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "8c06c832b9d14c9cb00e58190b2164a1"
format_stamp: "Formatted at 2026-08-10 14:31:46 on dist-test-slave-q9h9"
server_key: "76f3778562eed4119e82266b4f56b025"
server_key_iv: "5a6ae8c4aa9bfcf086d8eb7cd7aee913"
server_key_version: "encryptionkey@0"
I20260810 14:31:46.430861 11875 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:31:46.433990 11897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:46.435195 11875 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260810 14:31:46.435338 11875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "8c06c832b9d14c9cb00e58190b2164a1"
format_stamp: "Formatted at 2026-08-10 14:31:46 on dist-test-slave-q9h9"
server_key: "76f3778562eed4119e82266b4f56b025"
server_key_iv: "5a6ae8c4aa9bfcf086d8eb7cd7aee913"
server_key_version: "encryptionkey@0"
I20260810 14:31:46.435499 11875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:46.462144 11875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:46.465989 11875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:46.466297 11875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:46.467186 11875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:46.468417 11875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:46.468494 11875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:46.468614 11875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:46.468667 11875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:46.498703 11875 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:35953
I20260810 14:31:46.498762 12083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:35953 every 8 connection(s)
I20260810 14:31:46.499858 11875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:31:46.501897 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 11875
I20260810 14:31:46.502125 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:31:46.502620 6331 external_mini_cluster.cc:1428] Setting key 5cd95daf48c4fe3bb4a80c41657c9a0f
Aug 10 14:31:46 dist-test-slave-q9h9 krb5kdc[11477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372306, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:46.518298 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:46.502197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:35519 (local address 127.6.46.254:38563)
0810 14:31:46.502390 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:46.502397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:46.503484 (+ 1087us) server_negotiation.cc:407] Connection header received
0810 14:31:46.505311 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:46.505316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:46.505395 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:46.505494 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:46.506912 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:46.507850 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:46.509312 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:46.509852 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:46.509969 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:46.513123 (+ 3154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:46.513128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:46.513131 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:46.513163 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:46.515540 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:46.516191 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:46.516195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:46.516197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:46.516321 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:46.516723 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:46.516727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:46.516729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:46.516960 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:46.517088 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:46.517730 (+ 642us) server_negotiation.cc:299] Negotiation successful
0810 14:31:46.517950 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:31:46.519152 12087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:46.502574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:38563 (local address 127.6.46.194:35519)
0810 14:31:46.503337 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:46.503380 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:46.505063 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:46.505671 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:46.505679 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:46.506227 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:46.506754 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:46.506770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:46.507995 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:46.508007 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:46.509171 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:46.509187 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:46.510064 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:46.510071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:46.510264 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:46.510848 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:46.510869 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:46.512919 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:46.515674 (+ 2755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:46.515682 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:46.515696 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:46.516071 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:46.516423 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:46.516428 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:46.516432 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:46.516603 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:46.517079 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:46.517086 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:46.517373 (+ 287us) client_negotiation.cc:769] Sending connection context
0810 14:31:46.517585 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 14:31:46.518086 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":313,"threads_started":1}
I20260810 14:31:46.520680 12084 heartbeater.cc:347] Connected to a master server at 127.6.46.254:38563
I20260810 14:31:46.520937 12084 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:46.521579 12084 heartbeater.cc:511] Master 127.6.46.254:38563 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
I20260810 14:31:46.522949 11563 ts_manager.cc:194] Registered new tserver with Master: 8c06c832b9d14c9cb00e58190b2164a1 (127.6.46.194:35953)
I20260810 14:31:46.523825 11563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:35519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:31:46.545823 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38563
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:42285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:46.853060 12093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:46.853322 12093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:46.853451 12093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:46.860442 12093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:46.860554 12093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:46.860692 12093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:31:46.869807 12093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:42285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38563
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:46.870976 12093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:46.872180 12093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:46.884207 12101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:46.884233 12102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:46.884199 12106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:46.884938 12093 server_base.cc:1034] running on GCE node
I20260810 14:31:46.885432 12093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:46.886255 12093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:46.887449 12093 hybrid_clock.cc:648] HybridClock initialized: now 1786372306887404 us; error 38 us; skew 500 ppm
I20260810 14:31:46.887727 12093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:46 dist-test-slave-q9h9 krb5kdc[11477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372306, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:46.890611 12093 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:31:46.891934 12093 webserver.cc:460] Webserver started at http://127.6.46.195:40313/ using document root <none> and password file <none>
I20260810 14:31:46.892580 12093 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:46.892693 12093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:46.892956 12093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:46.895680 12093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "f705672f86a64c46babb69db82c21eac"
format_stamp: "Formatted at 2026-08-10 14:31:46 on dist-test-slave-q9h9"
server_key: "e7de088b2ed3bf4c27f3355e38e96750"
server_key_iv: "822db06fa3908e1b89b0ab010222c5d0"
server_key_version: "encryptionkey@0"
I20260810 14:31:46.896389 12093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "f705672f86a64c46babb69db82c21eac"
format_stamp: "Formatted at 2026-08-10 14:31:46 on dist-test-slave-q9h9"
server_key: "e7de088b2ed3bf4c27f3355e38e96750"
server_key_iv: "822db06fa3908e1b89b0ab010222c5d0"
server_key_version: "encryptionkey@0"
I20260810 14:31:46.900966 12093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 14:31:46.904152 12115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:46.905386 12093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 14:31:46.905537 12093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "f705672f86a64c46babb69db82c21eac"
format_stamp: "Formatted at 2026-08-10 14:31:46 on dist-test-slave-q9h9"
server_key: "e7de088b2ed3bf4c27f3355e38e96750"
server_key_iv: "822db06fa3908e1b89b0ab010222c5d0"
server_key_version: "encryptionkey@0"
I20260810 14:31:46.905717 12093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:46.920451 12093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:46.924070 12093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:46.924330 12093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:46.925247 12093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:46.926643 12093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:46.926721 12093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:46.926843 12093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:46.926919 12093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:46.957787 12093 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:44375
I20260810 14:31:46.957827 12293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:44375 every 8 connection(s)
I20260810 14:31:46.959177 12093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:31:46.961045 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 12093
I20260810 14:31:46.961261 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:31:46.961829 6331 external_mini_cluster.cc:1428] Setting key cdf422a104f995660dd91f7412c34d7a
Aug 10 14:31:46 dist-test-slave-q9h9 krb5kdc[11477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372306, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:46.978539 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:46.961672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:43249 (local address 127.6.46.254:38563)
0810 14:31:46.961837 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:46.961841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:46.963261 (+ 1420us) server_negotiation.cc:407] Connection header received
0810 14:31:46.965396 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:46.965402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:46.965489 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:46.965586 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:46.967079 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:46.967831 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:46.969311 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:46.969724 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:46.969833 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:46.973013 (+ 3180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:46.973018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:46.973022 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:46.973052 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:46.975463 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:46.976245 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:46.976251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:46.976254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:46.976374 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:46.976833 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:46.976838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:46.976842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:46.977176 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:46.977303 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:46.977961 (+ 658us) server_negotiation.cc:299] Negotiation successful
0810 14:31:46.978211 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:31:46.979420 12298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:46.962305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:38563 (local address 127.6.46.195:43249)
0810 14:31:46.963104 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:46.963151 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:46.965120 (+ 1969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:46.965832 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:46.965839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:46.966374 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:46.966915 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:46.966932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:46.967985 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:46.967990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:46.969172 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:46.969184 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:46.969935 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:46.969942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:46.970216 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:46.970772 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:46.970792 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:46.972806 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:46.975591 (+ 2785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:46.975600 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:46.975615 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:46.976150 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:46.976478 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:46.976483 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:46.976487 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:46.976736 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:46.977295 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:46.977303 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:46.977618 (+ 315us) client_negotiation.cc:769] Sending connection context
0810 14:31:46.977842 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 14:31:46.978357 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":304,"threads_started":1}
I20260810 14:31:46.980600 12294 heartbeater.cc:347] Connected to a master server at 127.6.46.254:38563
I20260810 14:31:46.980872 12294 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:46.981599 12294 heartbeater.cc:511] Master 127.6.46.254:38563 requested a full tablet report, sending...
I20260810 14:31:46.983021 11563 ts_manager.cc:194] Registered new tserver with Master: f705672f86a64c46babb69db82c21eac (127.6.46.195:44375)
I20260810 14:31:46.983807 11563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:43249
I20260810 14:31:46.989202 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 14:31:47 dist-test-slave-q9h9 krb5kdc[11477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372307, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 14:31:47 dist-test-slave-q9h9 krb5kdc[11477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372307, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:47.041324 11869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:47.027717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.6.46.254:38563)
0810 14:31:47.027946 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:47.027955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:47.028406 (+ 451us) server_negotiation.cc:407] Connection header received
0810 14:31:47.028571 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:47.028576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:47.028647 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:47.028729 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:47.030082 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:47.030870 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:47.032097 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:47.032549 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:47.032661 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:47.036248 (+ 3587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:47.036253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:47.036256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:47.036291 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:47.038915 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:47.039532 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:47.039535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:47.039538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:47.039696 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:47.040014 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:47.040017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:47.040019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:47.040331 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:47.040513 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:47.040795 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 14:31:47.041000 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 14:31:47.041918 11563 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43514
I20260810 14:31:47.043277 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11663
I20260810 14:31:47.043448 11866 heartbeater.cc:503] Master 127.6.46.254:38563 was elected leader, sending a full tablet report...
Aug 10 14:31:47 dist-test-slave-q9h9 krb5kdc[11477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372307, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:31:47.058521 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:47.044572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.6.46.193:44985)
0810 14:31:47.045584 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:47.045590 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:47.045614 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:31:47.045666 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:47.045672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:47.045854 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:47.045981 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:47.047167 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:47.048173 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:47.049354 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:47.049929 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:47.050071 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:47.053562 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:47.053572 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:47.053585 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:47.053615 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:47.056177 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:47.056809 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:47.056816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:47.056818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:47.056934 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:47.057230 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:47.057233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:47.057235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:47.057547 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:47.057776 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:47.057984 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 14:31:47.058241 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":703,"threads_started":1}
W20260810 14:31:47.059866 11856 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40802
W20260810 14:31:47.060976 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11663: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:31:47.061152 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 11663
I20260810 14:31:47.087823 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11875
Aug 10 14:31:47 dist-test-slave-q9h9 krb5kdc[11477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372307, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:31:47.102797 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:47.088661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.6.46.194:35953)
0810 14:31:47.089149 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:47.089155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:47.089180 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:31:47.089239 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:47.089245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:47.089444 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:47.089576 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:47.090893 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:47.091974 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:47.093212 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:47.093756 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:47.093862 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:47.097696 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:47.097707 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:47.097721 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:47.097756 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:47.100483 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:47.100998 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:47.101004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:47.101007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:47.101129 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:47.101395 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:47.101398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:47.101400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:47.101733 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:47.101972 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:47.102305 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 14:31:47.102502 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":219,"threads_started":1}
W20260810 14:31:47.104072 12077 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35222
W20260810 14:31:47.105126 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11875: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:31:47.105271 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 11875
I20260810 14:31:47.128386 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 12093
Aug 10 14:31:47 dist-test-slave-q9h9 krb5kdc[11477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372307, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:31:47.143822 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:47.129368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39096 (local address 127.6.46.195:44375)
0810 14:31:47.130006 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:47.130033 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:47.130070 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:31:47.130140 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:47.130147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:47.130351 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:47.130478 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:47.131782 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:47.132790 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:47.133959 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:47.134547 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:47.134654 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:47.138799 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:47.138812 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:47.138826 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:47.138861 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:47.141548 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:47.142113 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:47.142119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:47.142121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:47.142239 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:47.142509 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:47.142512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:47.142514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:47.142835 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:47.143064 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:47.143272 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 14:31:47.143462 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":331,"threads_started":1}
W20260810 14:31:47.145037 12285 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39096
W20260810 14:31:47.146116 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 12093: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:31:47.146251 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 12093
I20260810 14:31:47.168076 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11498
I20260810 14:31:47.169191 11611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:47.269928 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 11498
2026-08-10T14:31:47Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3594 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:31:47 dist-test-slave-q9h9 krb5kdc[12319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:31:47 dist-test-slave-q9h9 krb5kdc[12319](info): set up 2 sockets
Aug 10 14:31:47 dist-test-slave-q9h9 krb5kdc[12319](info): commencing operation
krb5kdc: starting...
W20260810 14:31:48.485020 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:31:48 dist-test-slave-q9h9 krb5kdc[12319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:31:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:31:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:31:48.664925 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:46037
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:46037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:48.972891 12339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:48.973230 12339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:48.973372 12339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:48.980298 12339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:31:48.980387 12339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:48.980432 12339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:48.980485 12339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:31:48.980527 12339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:31:48.989646 12339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:46037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:46037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:48.990790 12339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:48.991928 12339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:49.002919 12347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:49.003002 12351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:49.002964 12349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:49.003737 12339 server_base.cc:1034] running on GCE node
I20260810 14:31:49.004446 12339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:49.005710 12339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:49.006906 12339 hybrid_clock.cc:648] HybridClock initialized: now 1786372309006859 us; error 34 us; skew 500 ppm
I20260810 14:31:49.007217 12339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:49 dist-test-slave-q9h9 krb5kdc[12319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372309, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:49.014788 12339 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:31:49.016170 12339 webserver.cc:460] Webserver started at http://127.6.46.254:42851/ using document root <none> and password file <none>
I20260810 14:31:49.016753 12339 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:49.016850 12339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:49.017112 12339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:49.019872 12339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "69b9afb95d7144729d6922925732b292"
format_stamp: "Formatted at 2026-08-10 14:31:49 on dist-test-slave-q9h9"
server_key: "32c19be67cdf24f37db450a5e88c44b2"
server_key_iv: "9fdacd239fb5a87eab7a2bd017f10029"
server_key_version: "encryptionkey@0"
I20260810 14:31:49.020568 12339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "69b9afb95d7144729d6922925732b292"
format_stamp: "Formatted at 2026-08-10 14:31:49 on dist-test-slave-q9h9"
server_key: "32c19be67cdf24f37db450a5e88c44b2"
server_key_iv: "9fdacd239fb5a87eab7a2bd017f10029"
server_key_version: "encryptionkey@0"
I20260810 14:31:49.025233 12339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:31:49.028435 12362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:49.029709 12339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 14:31:49.029865 12339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "69b9afb95d7144729d6922925732b292"
format_stamp: "Formatted at 2026-08-10 14:31:49 on dist-test-slave-q9h9"
server_key: "32c19be67cdf24f37db450a5e88c44b2"
server_key_iv: "9fdacd239fb5a87eab7a2bd017f10029"
server_key_version: "encryptionkey@0"
I20260810 14:31:49.030055 12339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:49.046407 12339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:49.050305 12339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:49.050557 12339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:49.067260 12339 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:46037
I20260810 14:31:49.067293 12460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:46037 every 8 connection(s)
I20260810 14:31:49.068495 12339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:31:49.069092 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 12339
I20260810 14:31:49.069295 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:31:49.070050 6331 external_mini_cluster.cc:1428] Setting key 18ebb1cc56f50ed9579e7a8fc2a66e98
I20260810 14:31:49.072583 12462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 14:31:49 dist-test-slave-q9h9 krb5kdc[12319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:49.082828 12462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292: Bootstrap starting.
I20260810 14:31:49.085880 12462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:49.086899 12462 log.cc:824] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:49.089488 12462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292: No bootstrap required, opened a new log
I20260810 14:31:49.090428 12470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:49.071883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41672 (local address 127.6.46.254:46037)
0810 14:31:49.072641 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:49.072655 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:49.072703 (+ 48us) server_negotiation.cc:407] Connection header received
0810 14:31:49.074297 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:49.074316 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:49.074649 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:49.075005 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:49.076458 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:49.077759 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:49.078989 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:49.079616 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:49.079798 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:49.083685 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:49.083693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:49.083705 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:49.083740 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:49.086634 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:49.087277 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:49.087286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:49.087297 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:49.087529 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:49.087913 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:49.087918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:49.087921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:49.088413 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:49.088698 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:49.089088 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 14:31:49.089451 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":351,"threads_started":1}
I20260810 14:31:49.097397 12462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b9afb95d7144729d6922925732b292" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46037 } }
I20260810 14:31:49.097780 12462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:49.097874 12462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69b9afb95d7144729d6922925732b292, State: Initialized, Role: FOLLOWER
I20260810 14:31:49.098399 12462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b9afb95d7144729d6922925732b292" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46037 } }
I20260810 14:31:49.098615 12462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:31:49.098745 12462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:31:49.098905 12462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:49.101230 12462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b9afb95d7144729d6922925732b292" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46037 } }
I20260810 14:31:49.101677 12462 leader_election.cc:302] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69b9afb95d7144729d6922925732b292; no voters:
I20260810 14:31:49.102244 12462 leader_election.cc:288] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:31:49.102331 12475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:49.102952 12475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [term 1 LEADER]: Becoming Leader. State: Replica: 69b9afb95d7144729d6922925732b292, State: Running, Role: LEADER
I20260810 14:31:49.103420 12475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b9afb95d7144729d6922925732b292" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46037 } }
I20260810 14:31:49.104043 12462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:31:49.106443 12478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69b9afb95d7144729d6922925732b292. Latest consensus state: current_term: 1 leader_uuid: "69b9afb95d7144729d6922925732b292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b9afb95d7144729d6922925732b292" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46037 } } }
I20260810 14:31:49.106928 12478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:49.106467 12476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69b9afb95d7144729d6922925732b292" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69b9afb95d7144729d6922925732b292" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46037 } } }
I20260810 14:31:49.107492 12476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:49.108704 12487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:31:49.113095 12487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:31:49.120875 12487 catalog_manager.cc:1347] Generated new cluster ID: 715e147fd6de49139ca58f8e4bfce415
I20260810 14:31:49.120977 12487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:31:49.128886 12487 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:31:49.130134 12487 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:31:49.143258 12487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69b9afb95d7144729d6922925732b292: Generated new TSK 0
I20260810 14:31:49.143983 12487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:31:49.193019 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46037
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:49.504700 12506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:49.504971 12506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:49.505118 12506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:49.512022 12506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:49.512110 12506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:49.512247 12506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:31:49.521428 12506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:49.522559 12506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:49.523753 12506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:49.535800 12519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:49.535877 12516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:49.535876 12517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:49.536439 12506 server_base.cc:1034] running on GCE node
I20260810 14:31:49.537006 12506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:49.537827 12506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:49.539042 12506 hybrid_clock.cc:648] HybridClock initialized: now 1786372309538971 us; error 63 us; skew 500 ppm
I20260810 14:31:49.539332 12506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:49 dist-test-slave-q9h9 krb5kdc[12319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372309, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:49.542256 12506 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:31:49.543565 12506 webserver.cc:460] Webserver started at http://127.6.46.193:35857/ using document root <none> and password file <none>
I20260810 14:31:49.544174 12506 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:49.544299 12506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:49.544575 12506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:49.547328 12506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "ff95b3fe0870480b90c05c3fff5f6b32"
format_stamp: "Formatted at 2026-08-10 14:31:49 on dist-test-slave-q9h9"
server_key: "480ae9090be7afc06f6248fb02f51dd3"
server_key_iv: "5dc452161e33ea1b81087f6be4e322be"
server_key_version: "encryptionkey@0"
I20260810 14:31:49.548070 12506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "ff95b3fe0870480b90c05c3fff5f6b32"
format_stamp: "Formatted at 2026-08-10 14:31:49 on dist-test-slave-q9h9"
server_key: "480ae9090be7afc06f6248fb02f51dd3"
server_key_iv: "5dc452161e33ea1b81087f6be4e322be"
server_key_version: "encryptionkey@0"
I20260810 14:31:49.552830 12506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:31:49.556094 12526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:49.557263 12506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:31:49.557406 12506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "ff95b3fe0870480b90c05c3fff5f6b32"
format_stamp: "Formatted at 2026-08-10 14:31:49 on dist-test-slave-q9h9"
server_key: "480ae9090be7afc06f6248fb02f51dd3"
server_key_iv: "5dc452161e33ea1b81087f6be4e322be"
server_key_version: "encryptionkey@0"
I20260810 14:31:49.557595 12506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:49.584148 12506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:49.587836 12506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:49.588091 12506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:49.588974 12506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:49.590582 12506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:49.590654 12506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:49.590745 12506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:49.590787 12506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:49.620330 12506 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:46287
I20260810 14:31:49.620393 12712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:46287 every 8 connection(s)
I20260810 14:31:49.621491 12506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:31:49.631001 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 12506
I20260810 14:31:49.631228 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:31:49.632045 6331 external_mini_cluster.cc:1428] Setting key 6220c32321cd85ea454862d128df37f9
Aug 10 14:31:49 dist-test-slave-q9h9 krb5kdc[12319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372309, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:49.642661 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:49.623802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:42091 (local address 127.6.46.254:46037)
0810 14:31:49.624255 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:49.624260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:49.625208 (+ 948us) server_negotiation.cc:407] Connection header received
0810 14:31:49.627698 (+ 2490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:49.627704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:49.627793 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:49.627927 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:49.629884 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:49.631055 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:49.633405 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:49.634035 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:49.634161 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:49.637344 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:49.637349 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:49.637352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:49.637384 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:49.639728 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:49.640383 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:49.640389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:49.640393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:49.640511 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:49.640906 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:49.640912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:49.640915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:49.641201 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:49.641329 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:49.641979 (+ 650us) server_negotiation.cc:299] Negotiation successful
0810 14:31:49.642224 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":327,"thread_start_us":209,"threads_started":1}
I20260810 14:31:49.643477 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:49.624351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:46037 (local address 127.6.46.193:42091)
0810 14:31:49.625031 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:49.625077 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:49.627360 (+ 2283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:49.628169 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:49.628180 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:49.628887 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:49.629659 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:49.629683 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:49.631249 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:49.631255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:49.633218 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:49.633247 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:49.634251 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:49.634259 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:49.634498 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:49.635119 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:49.635141 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:49.637134 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:49.639853 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:49.639862 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:49.639877 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:49.640275 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:49.640620 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:49.640625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:49.640629 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:49.640801 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:49.641333 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:49.641341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:49.641633 (+ 292us) client_negotiation.cc:769] Sending connection context
0810 14:31:49.641866 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 14:31:49.642372 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":267,"threads_started":1}
I20260810 14:31:49.644867 12713 heartbeater.cc:347] Connected to a master server at 127.6.46.254:46037
I20260810 14:31:49.645217 12713 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:49.645985 12713 heartbeater.cc:511] Master 127.6.46.254:46037 requested a full tablet report, sending...
I20260810 14:31:49.648250 12403 ts_manager.cc:194] Registered new tserver with Master: ff95b3fe0870480b90c05c3fff5f6b32 (127.6.46.193:46287)
I20260810 14:31:49.650161 12403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:42091
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:31:49.677484 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46037
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:49.985426 12726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:49.985688 12726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:49.985824 12726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:49.992775 12726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:49.992872 12726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:49.993014 12726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:31:50.002185 12726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:50.003294 12726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:50.004556 12726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:50.016968 12736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:50.016899 12739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:50.016937 12737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:50.017732 12726 server_base.cc:1034] running on GCE node
I20260810 14:31:50.018285 12726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:50.019068 12726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:50.020249 12726 hybrid_clock.cc:648] HybridClock initialized: now 1786372310020212 us; error 37 us; skew 500 ppm
I20260810 14:31:50.020522 12726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:50 dist-test-slave-q9h9 krb5kdc[12319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372310, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:50.023531 12726 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:31:50.024766 12726 webserver.cc:460] Webserver started at http://127.6.46.194:34855/ using document root <none> and password file <none>
I20260810 14:31:50.025386 12726 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:50.025502 12726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:50.025786 12726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:50.028496 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "4907b289ab7a40f6b7ac33f695e482bf"
format_stamp: "Formatted at 2026-08-10 14:31:50 on dist-test-slave-q9h9"
server_key: "fd83c4d3e9eb0a6226307ea381dc35cc"
server_key_iv: "e7dc81aaa98a70a1f207418d49fb1596"
server_key_version: "encryptionkey@0"
I20260810 14:31:50.029219 12726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "4907b289ab7a40f6b7ac33f695e482bf"
format_stamp: "Formatted at 2026-08-10 14:31:50 on dist-test-slave-q9h9"
server_key: "fd83c4d3e9eb0a6226307ea381dc35cc"
server_key_iv: "e7dc81aaa98a70a1f207418d49fb1596"
server_key_version: "encryptionkey@0"
I20260810 14:31:50.034083 12726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 14:31:50.037251 12747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:50.038475 12726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260810 14:31:50.038619 12726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "4907b289ab7a40f6b7ac33f695e482bf"
format_stamp: "Formatted at 2026-08-10 14:31:50 on dist-test-slave-q9h9"
server_key: "fd83c4d3e9eb0a6226307ea381dc35cc"
server_key_iv: "e7dc81aaa98a70a1f207418d49fb1596"
server_key_version: "encryptionkey@0"
I20260810 14:31:50.038793 12726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:50.052664 12726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:50.056478 12726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:50.056730 12726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:50.057574 12726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:50.058791 12726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:50.058871 12726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:50.058979 12726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:50.059031 12726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:50.088881 12726 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:40051
I20260810 14:31:50.088898 12936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:40051 every 8 connection(s)
I20260810 14:31:50.090057 12726 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:31:50.092991 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 12726
I20260810 14:31:50.093169 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:31:50.093606 6331 external_mini_cluster.cc:1428] Setting key d7a9eef9c3c120480c1a5489abf61fe6
Aug 10 14:31:50 dist-test-slave-q9h9 krb5kdc[12319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372310, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:50.109817 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:50.092509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:32931 (local address 127.6.46.254:46037)
0810 14:31:50.092746 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:50.092752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:50.093871 (+ 1119us) server_negotiation.cc:407] Connection header received
0810 14:31:50.095963 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:50.095969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:50.096047 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:50.096153 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:50.097657 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.098619 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.100132 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.100694 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.100814 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:50.104178 (+ 3364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:50.104183 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:50.104186 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:50.104218 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:50.106782 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.107607 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.107611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.107614 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.107754 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.108241 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.108244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.108246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.108510 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:50.108682 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:50.109396 (+ 714us) server_negotiation.cc:299] Negotiation successful
0810 14:31:50.109569 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 14:31:50.111083 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:50.092905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:46037 (local address 127.6.46.194:32931)
0810 14:31:50.093699 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:50.093755 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:50.095709 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:50.096436 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:50.096444 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:50.096933 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:50.097472 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:50.097490 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.098762 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.098766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:50.099985 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:50.099997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.100893 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.100898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:50.101082 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:50.101689 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:50.101710 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:50.103972 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:50.106922 (+ 2950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:50.106931 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:50.106945 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:50.107500 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:50.107860 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:50.107865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:50.107868 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:50.108137 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:50.108664 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:50.108671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:50.109027 (+ 356us) client_negotiation.cc:769] Sending connection context
0810 14:31:50.109289 (+ 262us) client_negotiation.cc:240] Negotiation successful
0810 14:31:50.109820 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":338,"threads_started":1}
I20260810 14:31:50.112483 12939 heartbeater.cc:347] Connected to a master server at 127.6.46.254:46037
I20260810 14:31:50.112766 12939 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:50.113448 12939 heartbeater.cc:511] Master 127.6.46.254:46037 requested a full tablet report, sending...
I20260810 14:31:50.114930 12403 ts_manager.cc:194] Registered new tserver with Master: 4907b289ab7a40f6b7ac33f695e482bf (127.6.46.194:40051)
I20260810 14:31:50.116026 12403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:32931
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:31:50.139222 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46037
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:50.460978 12948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:50.461360 12948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:50.461519 12948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:50.472658 12948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:50.472787 12948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:50.472975 12948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:31:50.488013 12948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:50.489634 12948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:50.491449 12948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:50.506486 12955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:50.507000 12958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:50.507282 12948 server_base.cc:1034] running on GCE node
W20260810 14:31:50.514325 12956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:50.515051 12948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:50.516018 12948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:50.517247 12948 hybrid_clock.cc:648] HybridClock initialized: now 1786372310517178 us; error 56 us; skew 500 ppm
I20260810 14:31:50.517566 12948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:50 dist-test-slave-q9h9 krb5kdc[12319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372310, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:50.520895 12948 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:31:50.522360 12948 webserver.cc:460] Webserver started at http://127.6.46.195:40093/ using document root <none> and password file <none>
I20260810 14:31:50.523018 12948 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:50.523149 12948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:50.523448 12948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:50.526448 12948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "26b18d8d1168469f8079c4e1f0a07344"
format_stamp: "Formatted at 2026-08-10 14:31:50 on dist-test-slave-q9h9"
server_key: "b73f85a58281beae7aa8af84c2a885f3"
server_key_iv: "a65309bb99127b14b138a66082f247de"
server_key_version: "encryptionkey@0"
I20260810 14:31:50.527242 12948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "26b18d8d1168469f8079c4e1f0a07344"
format_stamp: "Formatted at 2026-08-10 14:31:50 on dist-test-slave-q9h9"
server_key: "b73f85a58281beae7aa8af84c2a885f3"
server_key_iv: "a65309bb99127b14b138a66082f247de"
server_key_version: "encryptionkey@0"
I20260810 14:31:50.532732 12948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 14:31:50.536605 12970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:50.538055 12948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:31:50.538223 12948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "26b18d8d1168469f8079c4e1f0a07344"
format_stamp: "Formatted at 2026-08-10 14:31:50 on dist-test-slave-q9h9"
server_key: "b73f85a58281beae7aa8af84c2a885f3"
server_key_iv: "a65309bb99127b14b138a66082f247de"
server_key_version: "encryptionkey@0"
I20260810 14:31:50.538410 12948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:50.563303 12948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:50.567582 12948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:50.567878 12948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:50.568842 12948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:50.570247 12948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:50.570333 12948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:50.570461 12948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:50.570519 12948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:50.605396 12948 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:38635
I20260810 14:31:50.605481 13147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:38635 every 8 connection(s)
I20260810 14:31:50.606763 12948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:31:50.616096 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 12948
I20260810 14:31:50.616278 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:31:50.616878 6331 external_mini_cluster.cc:1428] Setting key 9d15af8fa8ab9484508285aee882afd9
Aug 10 14:31:50 dist-test-slave-q9h9 krb5kdc[12319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372310, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:50.632794 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:50.609705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:33617 (local address 127.6.46.254:46037)
0810 14:31:50.609927 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:50.609934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:50.611273 (+ 1339us) server_negotiation.cc:407] Connection header received
0810 14:31:50.613879 (+ 2606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:50.613884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:50.613961 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:50.614103 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:50.616217 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.617315 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.619731 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.620381 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.620488 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:50.625301 (+ 4813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:50.625308 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:50.625314 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:50.625365 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:50.628936 (+ 3571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.629721 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.629726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.629730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.629900 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.630494 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.630500 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.630503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.630934 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:50.631094 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:50.632121 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0810 14:31:50.632399 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:31:50.633847 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:50.610218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:46037 (local address 127.6.46.195:33617)
0810 14:31:50.611085 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:50.611137 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:50.613553 (+ 2416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:50.614354 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:50.614367 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:50.615188 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:50.615996 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:50.616023 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.617483 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.617488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:50.619542 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:50.619560 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.620890 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.620898 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:50.621197 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:50.621970 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:50.622008 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:50.625043 (+ 3035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:50.629110 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:50.629118 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:50.629129 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:50.629586 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:50.630069 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:50.630075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:50.630079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:50.630355 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:50.631103 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:50.631111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:50.631565 (+ 454us) client_negotiation.cc:769] Sending connection context
0810 14:31:50.631988 (+ 423us) client_negotiation.cc:240] Negotiation successful
0810 14:31:50.632613 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":336,"threads_started":1}
I20260810 14:31:50.635196 13150 heartbeater.cc:347] Connected to a master server at 127.6.46.254:46037
I20260810 14:31:50.635551 13150 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:50.636474 13150 heartbeater.cc:511] Master 127.6.46.254:46037 requested a full tablet report, sending...
I20260810 14:31:50.638130 12403 ts_manager.cc:194] Registered new tserver with Master: 26b18d8d1168469f8079c4e1f0a07344 (127.6.46.195:38635)
I20260810 14:31:50.638959 12403 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:33617
I20260810 14:31:50.645112 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:31:50.653514 12713 heartbeater.cc:503] Master 127.6.46.254:46037 was elected leader, sending a full tablet report...
Aug 10 14:31:50 dist-test-slave-q9h9 krb5kdc[12319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:31:50 dist-test-slave-q9h9 krb5kdc[12319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:31:50.695722 13159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:50.682045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.6.46.193:46287)
0810 14:31:50.682649 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:50.682656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:50.682683 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:31:50.682743 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:50.682749 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:50.682960 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:50.683136 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:50.684423 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.685426 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.686651 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.687205 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.687298 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:50.690852 (+ 3554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:50.690864 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:50.690877 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:50.690912 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:50.693420 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.693971 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.693977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.693979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.694145 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.694433 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.694436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.694438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.694768 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:50.694983 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:50.695182 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 14:31:50.695368 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":309,"threads_started":1}
Aug 10 14:31:50 dist-test-slave-q9h9 krb5kdc[12319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:50.712473 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:50.699627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.6.46.254:46037)
0810 14:31:50.699832 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:50.699838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:50.699960 (+ 122us) server_negotiation.cc:407] Connection header received
0810 14:31:50.700110 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:50.700114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:50.700180 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:50.700261 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:50.701392 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.702175 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.703365 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.703777 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.703887 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:50.707378 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:50.707383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:50.707386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:50.707419 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:50.710142 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.710697 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.710703 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.710707 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.710835 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.711161 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.711166 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.711170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.711469 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:50.711704 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:50.711967 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 14:31:50.712171 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 14:31:50.712992 12403 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41676
I20260810 14:31:50.714008 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 12506
I20260810 14:31:50.725636 13159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:50.714894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.6.46.193:46287)
0810 14:31:50.715030 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:50.715034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:50.715300 (+ 266us) server_negotiation.cc:407] Connection header received
0810 14:31:50.715422 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:50.715426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:50.715509 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:50.715581 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:50.716754 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.717412 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.718745 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.719181 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.719280 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:50.721349 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:50.721354 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:50.721357 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:50.721389 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:50.723689 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.724230 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.724233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.724235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.724342 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.724653 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.724656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.724658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.724885 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:50.725052 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:50.725266 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 14:31:50.725410 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:31:50.726202 12706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:50.806443 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 12506
I20260810 14:31:50.827912 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 12726
Aug 10 14:31:50 dist-test-slave-q9h9 krb5kdc[12319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:31:50.843580 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:50.828920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51586 (local address 127.6.46.194:40051)
0810 14:31:50.829457 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:50.829464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:50.829500 (+ 36us) server_negotiation.cc:407] Connection header received
0810 14:31:50.829560 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:50.829567 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:50.829778 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:50.829905 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:50.831282 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.832342 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.833696 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.834379 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.834477 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:50.838341 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:50.838351 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:50.838364 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:50.838396 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:50.841052 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.841615 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.841622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.841624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.841763 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.842194 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.842199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.842203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.842552 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:50.842834 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:50.843031 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 14:31:50.843276 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":255,"threads_started":1}
I20260810 14:31:50.844801 12929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:50.925496 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 12726
I20260810 14:31:50.943379 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 12948
Aug 10 14:31:50 dist-test-slave-q9h9 krb5kdc[12319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:31:50.958324 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:50.944246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57528 (local address 127.6.46.195:38635)
0810 14:31:50.944872 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:50.944879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:50.944908 (+ 29us) server_negotiation.cc:407] Connection header received
0810 14:31:50.944969 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:50.944976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:50.945182 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:50.945312 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:50.946577 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.947583 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.948753 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:50.949290 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:50.949382 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:50.953361 (+ 3979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:50.953373 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:50.953387 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:50.953421 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:50.955957 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.956512 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.956518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.956520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.956632 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:50.956889 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:50.956892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:50.956894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:50.957209 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:50.957432 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:50.957612 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 14:31:50.957819 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":288,"threads_started":1}
I20260810 14:31:50.959533 13142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:51.039217 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 12948
I20260810 14:31:51.062471 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 12339
I20260810 14:31:51.063602 12452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:51.162793 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 12339
2026-08-10T14:31:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3891 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:31:51 dist-test-slave-q9h9 krb5kdc[13185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:31:51 dist-test-slave-q9h9 krb5kdc[13185](info): set up 2 sockets
Aug 10 14:31:51 dist-test-slave-q9h9 krb5kdc[13185](info): commencing operation
krb5kdc: starting...
W20260810 14:31:52.360196 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.148s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:31:52 dist-test-slave-q9h9 krb5kdc[13185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:31:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:31:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:31:52.548902 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:45391
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:45391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:52.862671 13201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:52.862931 13201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:52.863076 13201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:52.870172 13201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:31:52.870275 13201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:52.870353 13201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:52.870430 13201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:31:52.870499 13201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:31:52.880657 13201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:45391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:45391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:52.881891 13201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:52.883157 13201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:52.893895 13208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:52.893926 13211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:52.894001 13213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:52.894735 13201 server_base.cc:1034] running on GCE node
I20260810 14:31:52.895430 13201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:52.896679 13201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:52.897878 13201 hybrid_clock.cc:648] HybridClock initialized: now 1786372312897835 us; error 37 us; skew 500 ppm
I20260810 14:31:52.898231 13201 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:52 dist-test-slave-q9h9 krb5kdc[13185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372312, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:52.901132 13201 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:31:52.902246 13201 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:52.902334 13201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:52.902609 13201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:52.905419 13201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "61f5c9666558466b9c599204c18c1256"
format_stamp: "Formatted at 2026-08-10 14:31:52 on dist-test-slave-q9h9"
server_key: "609bb664bbd08e5accf672cf4fed72b1"
server_key_iv: "6eca4d6e43fde0e24f04fb6a5cff31c7"
server_key_version: "encryptionkey@0"
I20260810 14:31:52.906213 13201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "61f5c9666558466b9c599204c18c1256"
format_stamp: "Formatted at 2026-08-10 14:31:52 on dist-test-slave-q9h9"
server_key: "609bb664bbd08e5accf672cf4fed72b1"
server_key_iv: "6eca4d6e43fde0e24f04fb6a5cff31c7"
server_key_version: "encryptionkey@0"
I20260810 14:31:52.910974 13201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:31:52.914212 13222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:52.915450 13201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:31:52.915611 13201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "61f5c9666558466b9c599204c18c1256"
format_stamp: "Formatted at 2026-08-10 14:31:52 on dist-test-slave-q9h9"
server_key: "609bb664bbd08e5accf672cf4fed72b1"
server_key_iv: "6eca4d6e43fde0e24f04fb6a5cff31c7"
server_key_version: "encryptionkey@0"
I20260810 14:31:52.915793 13201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:52.930966 13201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:52.934453 13201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:52.934662 13201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:52.951465 13201 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:45391
I20260810 14:31:52.951485 13305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:45391 every 8 connection(s)
I20260810 14:31:52.952265 13201 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:31:52.955068 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 13201
I20260810 14:31:52.955279 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:31:52.955744 13307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:52.955910 6331 external_mini_cluster.cc:1428] Setting key 4ab19c4e91faa470e6dc58e565c7589b
I20260810 14:31:52.969297 13307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256: Bootstrap starting.
Aug 10 14:31:52 dist-test-slave-q9h9 krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:52.972245 13307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:52.973243 13307 log.cc:824] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:52.975728 13307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256: No bootstrap required, opened a new log
I20260810 14:31:52.977281 13310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:52.957721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59976 (local address 127.6.46.254:45391)
0810 14:31:52.958523 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:52.958533 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:52.958569 (+ 36us) server_negotiation.cc:407] Connection header received
0810 14:31:52.960114 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:52.960135 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:52.960471 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:52.960795 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:52.962607 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:52.963843 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:52.965534 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:52.966267 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:52.966405 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:52.970849 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:52.970857 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:52.970870 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:52.970903 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:52.973633 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:52.974300 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:52.974306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:52.974311 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:52.974469 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:52.974834 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:52.974837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:52.974840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:52.975322 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:52.975587 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:52.976007 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 14:31:52.976391 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":380,"threads_started":1}
I20260810 14:31:52.983232 13307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f5c9666558466b9c599204c18c1256" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45391 } }
I20260810 14:31:52.983553 13307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:52.983639 13307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61f5c9666558466b9c599204c18c1256, State: Initialized, Role: FOLLOWER
I20260810 14:31:52.984153 13307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f5c9666558466b9c599204c18c1256" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45391 } }
I20260810 14:31:52.984392 13307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:31:52.984514 13307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:31:52.984676 13307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:52.986958 13307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f5c9666558466b9c599204c18c1256" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45391 } }
I20260810 14:31:52.987411 13307 leader_election.cc:302] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61f5c9666558466b9c599204c18c1256; no voters:
I20260810 14:31:52.987927 13307 leader_election.cc:288] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:31:52.988030 13318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:52.988641 13318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [term 1 LEADER]: Becoming Leader. State: Replica: 61f5c9666558466b9c599204c18c1256, State: Running, Role: LEADER
I20260810 14:31:52.989117 13318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f5c9666558466b9c599204c18c1256" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45391 } }
I20260810 14:31:52.989774 13307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:31:52.991698 13319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61f5c9666558466b9c599204c18c1256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f5c9666558466b9c599204c18c1256" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45391 } } }
I20260810 14:31:52.991966 13319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:52.993391 13320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61f5c9666558466b9c599204c18c1256. Latest consensus state: current_term: 1 leader_uuid: "61f5c9666558466b9c599204c18c1256" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61f5c9666558466b9c599204c18c1256" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45391 } } }
I20260810 14:31:52.993619 13320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:52.994194 13324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:31:52.998823 13324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:31:53.006820 13324 catalog_manager.cc:1347] Generated new cluster ID: d79f93b056bb466189eb97838a1b62f8
I20260810 14:31:53.006918 13324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:31:53.034379 13324 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:31:53.036206 13324 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:31:53.043419 13324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256: Generated new TSK 0
I20260810 14:31:53.044346 13324 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:31:53.094617 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:45391
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:53.407207 13351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:53.407444 13351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:53.407537 13351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:53.414477 13351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:53.414567 13351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:53.414685 13351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:31:53.423949 13351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:45391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:53.425038 13351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:53.426293 13351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:53.438091 13359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:53.438210 13358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:53.438185 13362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:53.438915 13351 server_base.cc:1034] running on GCE node
I20260810 14:31:53.439433 13351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:53.440310 13351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:53.441502 13351 hybrid_clock.cc:648] HybridClock initialized: now 1786372313441451 us; error 48 us; skew 500 ppm
I20260810 14:31:53.441803 13351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:53 dist-test-slave-q9h9 krb5kdc[13185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:53.444810 13351 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:31:53.445904 13351 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:53.446002 13351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:53.446331 13351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:53.449087 13351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "289154ae263f403ca5faba66ce1c66c5"
format_stamp: "Formatted at 2026-08-10 14:31:53 on dist-test-slave-q9h9"
server_key: "6efa4873c6c82c7d1745ca2c0b708ae8"
server_key_iv: "58e0ff3ba8e4f1efefaf6a0a2c96cd00"
server_key_version: "encryptionkey@0"
I20260810 14:31:53.449865 13351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "289154ae263f403ca5faba66ce1c66c5"
format_stamp: "Formatted at 2026-08-10 14:31:53 on dist-test-slave-q9h9"
server_key: "6efa4873c6c82c7d1745ca2c0b708ae8"
server_key_iv: "58e0ff3ba8e4f1efefaf6a0a2c96cd00"
server_key_version: "encryptionkey@0"
I20260810 14:31:53.454584 13351 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 14:31:53.457795 13372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:53.458999 13351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:31:53.459141 13351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "289154ae263f403ca5faba66ce1c66c5"
format_stamp: "Formatted at 2026-08-10 14:31:53 on dist-test-slave-q9h9"
server_key: "6efa4873c6c82c7d1745ca2c0b708ae8"
server_key_iv: "58e0ff3ba8e4f1efefaf6a0a2c96cd00"
server_key_version: "encryptionkey@0"
I20260810 14:31:53.459328 13351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:53.483516 13351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:53.487058 13351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:53.487289 13351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:53.488160 13351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:53.489355 13351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:53.489450 13351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:53.489553 13351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:53.489631 13351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:53.519469 13351 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:42453
I20260810 14:31:53.519563 13557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:42453 every 8 connection(s)
I20260810 14:31:53.520253 13351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:31:53.522229 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 13351
I20260810 14:31:53.522437 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:31:53.523234 6331 external_mini_cluster.cc:1428] Setting key 44d06259ece206573d6fe006215aa0c2
Aug 10 14:31:53 dist-test-slave-q9h9 krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:53.538817 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:53.522462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:55927 (local address 127.6.46.254:45391)
0810 14:31:53.522874 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:53.522879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:53.524014 (+ 1135us) server_negotiation.cc:407] Connection header received
0810 14:31:53.526108 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:53.526114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:53.526187 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:53.526277 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:53.527659 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:53.528368 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:53.529836 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:53.530273 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:53.530382 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:53.533474 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:53.533478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:53.533481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:53.533512 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:53.535895 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:53.536664 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:53.536667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:53.536670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:53.536798 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:53.537257 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:53.537260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:53.537262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:53.537544 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:53.537650 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:53.538408 (+ 758us) server_negotiation.cc:299] Negotiation successful
0810 14:31:53.538564 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":214,"threads_started":1}
I20260810 14:31:53.539999 13563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:53.523083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:45391 (local address 127.6.46.193:55927)
0810 14:31:53.523833 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:53.523891 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:53.525842 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:53.526500 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:53.526507 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:53.526985 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:53.527494 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:53.527511 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:53.528508 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:53.528512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:53.529703 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:53.529716 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:53.530482 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:53.530490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:53.530692 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:53.531265 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:53.531286 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:53.533277 (+ 1991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:53.536044 (+ 2767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:53.536052 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:53.536065 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:53.536571 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:53.536924 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:53.536928 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:53.536931 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:53.537158 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:53.537672 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:53.537679 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:53.538077 (+ 398us) client_negotiation.cc:769] Sending connection context
0810 14:31:53.538346 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 14:31:53.538795 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":327,"threads_started":1}
I20260810 14:31:53.541383 13558 heartbeater.cc:347] Connected to a master server at 127.6.46.254:45391
I20260810 14:31:53.541640 13558 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:53.542250 13558 heartbeater.cc:511] Master 127.6.46.254:45391 requested a full tablet report, sending...
I20260810 14:31:53.544246 13250 ts_manager.cc:194] Registered new tserver with Master: 289154ae263f403ca5faba66ce1c66c5 (127.6.46.193:42453)
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
I20260810 14:31:53.545608 13250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:55927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:31:53.572620 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:45391
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:53.884006 13567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:53.884258 13567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:53.884392 13567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:53.891511 13567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:53.891621 13567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:53.891765 13567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:31:53.901024 13567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:45391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:53.902194 13567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:53.903386 13567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:53.915089 13578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:53.915396 13580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:53.915627 13577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:53.916260 13567 server_base.cc:1034] running on GCE node
I20260810 14:31:53.916802 13567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:53.917634 13567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:53.918823 13567 hybrid_clock.cc:648] HybridClock initialized: now 1786372313918777 us; error 46 us; skew 500 ppm
I20260810 14:31:53.919117 13567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:53 dist-test-slave-q9h9 krb5kdc[13185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:53.922052 13567 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:31:53.923090 13567 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:53.923177 13567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:53.923435 13567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:53.926115 13567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "02f3f4e5859d47908aa332ab9931d1fb"
format_stamp: "Formatted at 2026-08-10 14:31:53 on dist-test-slave-q9h9"
server_key: "f9929d6f0550d4ab323c88885a8f62ba"
server_key_iv: "1433a29e22958cf248904091446fa919"
server_key_version: "encryptionkey@0"
I20260810 14:31:53.926857 13567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "02f3f4e5859d47908aa332ab9931d1fb"
format_stamp: "Formatted at 2026-08-10 14:31:53 on dist-test-slave-q9h9"
server_key: "f9929d6f0550d4ab323c88885a8f62ba"
server_key_iv: "1433a29e22958cf248904091446fa919"
server_key_version: "encryptionkey@0"
I20260810 14:31:53.931512 13567 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 14:31:53.934852 13592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:53.936034 13567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 14:31:53.936162 13567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "02f3f4e5859d47908aa332ab9931d1fb"
format_stamp: "Formatted at 2026-08-10 14:31:53 on dist-test-slave-q9h9"
server_key: "f9929d6f0550d4ab323c88885a8f62ba"
server_key_iv: "1433a29e22958cf248904091446fa919"
server_key_version: "encryptionkey@0"
I20260810 14:31:53.936328 13567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:53.957304 13567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:53.961186 13567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:53.961412 13567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:53.962360 13567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:53.963690 13567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:53.963775 13567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:53.963878 13567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:53.963944 13567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:53.993422 13567 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:38933
I20260810 14:31:53.993481 13793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:38933 every 8 connection(s)
I20260810 14:31:53.994277 13567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:31:53.999514 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 13567
I20260810 14:31:53.999692 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:31:54.000224 6331 external_mini_cluster.cc:1428] Setting key d3b8b7452f7afe811816a2a270a54890
Aug 10 14:31:54 dist-test-slave-q9h9 krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372313, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:54.012689 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:53.996526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:34313 (local address 127.6.46.254:45391)
0810 14:31:53.996688 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:53.996692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:53.997721 (+ 1029us) server_negotiation.cc:407] Connection header received
0810 14:31:53.999633 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:53.999640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:53.999727 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:53.999841 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:54.001420 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.002298 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.003756 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.004136 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.004243 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.007567 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:54.007572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:54.007575 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:54.007604 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:54.009897 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:54.010569 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:54.010573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:54.010576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:54.010709 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:54.011085 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:54.011088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:54.011090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:54.011371 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:54.011521 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:54.012113 (+ 592us) server_negotiation.cc:299] Negotiation successful
0810 14:31:54.012364 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:31:54.013536 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:53.996914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:45391 (local address 127.6.46.194:34313)
0810 14:31:53.997561 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:53.997604 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:53.999333 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:54.000081 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:54.000088 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:54.000618 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:54.001260 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.001277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.002434 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.002439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.003623 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.003635 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.004340 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.004348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.004541 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.005125 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:54.005147 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:54.007371 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:54.010037 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:54.010047 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:54.010061 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:54.010462 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:54.010808 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:54.010813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:54.010817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:54.010976 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:54.011490 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:54.011497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:54.011782 (+ 285us) client_negotiation.cc:769] Sending connection context
0810 14:31:54.011996 (+ 214us) client_negotiation.cc:240] Negotiation successful
0810 14:31:54.012479 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":262,"threads_started":1}
I20260810 14:31:54.014895 13794 heartbeater.cc:347] Connected to a master server at 127.6.46.254:45391
I20260810 14:31:54.015210 13794 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:54.015869 13794 heartbeater.cc:511] Master 127.6.46.254:45391 requested a full tablet report, sending...
I20260810 14:31:54.017354 13250 ts_manager.cc:194] Registered new tserver with Master: 02f3f4e5859d47908aa332ab9931d1fb (127.6.46.194:38933)
I20260810 14:31:54.018456 13250 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:34313
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:31:54.045125 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:45391
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:46341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:31:54.355301 13799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:54.355558 13799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:54.355682 13799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:54.363143 13799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:54.363246 13799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:54.363406 13799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:31:54.372771 13799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:46341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:45391
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:54.373962 13799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:54.375293 13799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:54.387408 13812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:54.387445 13809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:54.387485 13808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:54.388476 13799 server_base.cc:1034] running on GCE node
I20260810 14:31:54.389063 13799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:54.389964 13799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:54.391194 13799 hybrid_clock.cc:648] HybridClock initialized: now 1786372314391140 us; error 48 us; skew 500 ppm
I20260810 14:31:54.391484 13799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:31:54 dist-test-slave-q9h9 krb5kdc[13185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372314, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:31:54.398720 13799 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:31:54.399840 13799 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:54.399946 13799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:54.400269 13799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:54.403210 13799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "d05f51d2d5db414aa890c4ca2f01c8cd"
format_stamp: "Formatted at 2026-08-10 14:31:54 on dist-test-slave-q9h9"
server_key: "efa9a335862d5a74e1f6051bb2ca1474"
server_key_iv: "cac026f668d844d33f12ec0a6e40ab8b"
server_key_version: "encryptionkey@0"
I20260810 14:31:54.404223 13799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "d05f51d2d5db414aa890c4ca2f01c8cd"
format_stamp: "Formatted at 2026-08-10 14:31:54 on dist-test-slave-q9h9"
server_key: "efa9a335862d5a74e1f6051bb2ca1474"
server_key_iv: "cac026f668d844d33f12ec0a6e40ab8b"
server_key_version: "encryptionkey@0"
I20260810 14:31:54.409309 13799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 14:31:54.412674 13818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:54.413892 13799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:31:54.414041 13799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "d05f51d2d5db414aa890c4ca2f01c8cd"
format_stamp: "Formatted at 2026-08-10 14:31:54 on dist-test-slave-q9h9"
server_key: "efa9a335862d5a74e1f6051bb2ca1474"
server_key_iv: "cac026f668d844d33f12ec0a6e40ab8b"
server_key_version: "encryptionkey@0"
I20260810 14:31:54.414243 13799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:54.465173 13799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:54.469192 13799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:54.469419 13799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:54.470350 13799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:54.471675 13799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:54.471793 13799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:54.471887 13799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:54.471946 13799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:54.502559 13799 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:44927
I20260810 14:31:54.502617 14005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:44927 every 8 connection(s)
I20260810 14:31:54.503376 13799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:31:54.512725 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 13799
I20260810 14:31:54.512899 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:31:54.513427 6331 external_mini_cluster.cc:1428] Setting key c583891fac07705ecbdc2f3198e03e5e
Aug 10 14:31:54 dist-test-slave-q9h9 krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372314, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:31:54.523722 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.505621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:44513 (local address 127.6.46.254:45391)
0810 14:31:54.505790 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:54.505794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:54.506963 (+ 1169us) server_negotiation.cc:407] Connection header received
0810 14:31:54.508854 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:54.508862 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:54.508952 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:54.509057 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:54.510706 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.511462 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.513330 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.513804 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.513915 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.517629 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:54.517642 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:54.517646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:54.517690 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:54.520315 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:54.521203 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:54.521207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:54.521210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:54.521338 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:54.521921 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:54.521927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:54.521931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:54.522388 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:54.522564 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:54.523320 (+ 756us) server_negotiation.cc:299] Negotiation successful
0810 14:31:54.523477 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:31:54.524958 14009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.506083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:45391 (local address 127.6.46.195:44513)
0810 14:31:54.506800 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:54.506845 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:54.508477 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:54.509314 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:54.509324 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:54.509869 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:54.510531 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.510551 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.511610 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.511613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.513138 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.513167 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.514035 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.514043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.514279 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.514994 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:31:54.515024 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:54.517439 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:54.520442 (+ 3003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:54.520450 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:54.520463 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:54.521052 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:54.521452 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:31:54.521456 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:31:54.521460 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:31:54.521728 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:31:54.522553 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:54.522561 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:54.522967 (+ 406us) client_negotiation.cc:769] Sending connection context
0810 14:31:54.523234 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 14:31:54.523726 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":275,"threads_started":1}
I20260810 14:31:54.526149 14006 heartbeater.cc:347] Connected to a master server at 127.6.46.254:45391
I20260810 14:31:54.526399 14006 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:54.526952 14006 heartbeater.cc:511] Master 127.6.46.254:45391 requested a full tablet report, sending...
I20260810 14:31:54.528439 13249 ts_manager.cc:194] Registered new tserver with Master: d05f51d2d5db414aa890c4ca2f01c8cd (127.6.46.195:44927)
I20260810 14:31:54.529263 13249 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:44513
I20260810 14:31:54.540370 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:31:54.548348 13558 heartbeater.cc:503] Master 127.6.46.254:45391 was elected leader, sending a full tablet report...
I20260810 14:31:54.556180 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.544691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.6.46.254:45391)
0810 14:31:54.544847 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:54.544851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:54.545328 (+ 477us) server_negotiation.cc:407] Connection header received
0810 14:31:54.545453 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:54.545456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:54.545509 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:54.545585 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:54.546978 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.547673 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.549061 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.549518 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.549635 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.551737 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:54.551742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:54.551745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:54.551772 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:54.553932 (+ 2160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:54.554550 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:54.554555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:54.554557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:54.554681 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:54.555063 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:54.555066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:54.555068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:54.555340 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:54.555514 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:54.555780 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 14:31:54.555923 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:31:54.559876 13249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:31:54.566550 13249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:31:54.608302 14028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.577775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:38933 (local address 127.0.0.1:43200)
0810 14:31:54.598969 (+ 21194us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:54.599022 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:54.600045 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:54.601122 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:54.601130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:54.601163 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:54.601615 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.601629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.603373 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.603379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.605298 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.605311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.607340 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.607349 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.607526 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.607572 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 14:31:54.607763 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 14:31:54.607951 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20899,"thread_start_us":23033,"threads_started":1}
I20260810 14:31:54.608363 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.577994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43200 (local address 127.6.46.194:38933)
0810 14:31:54.590306 (+ 12312us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:54.590316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:54.599172 (+ 8856us) server_negotiation.cc:407] Connection header received
0810 14:31:54.600349 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:54.600358 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:54.600636 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:54.600806 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:54.601836 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.603203 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.605464 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.606860 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.606980 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.607191 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:54.607828 (+ 637us) server_negotiation.cc:299] Negotiation successful
0810 14:31:54.608022 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11936,"thread_start_us":11483,"threads_started":1}
I20260810 14:31:54.609220 14022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.577936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:44927 (local address 127.0.0.1:51102)
0810 14:31:54.600521 (+ 22585us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:54.600545 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:54.600684 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:54.602127 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:54.602132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:54.602159 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:54.602523 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.602533 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.604323 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.604328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.606273 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.606286 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.608539 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.608546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.608669 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.608691 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:31:54.608779 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 14:31:54.608893 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22438,"thread_start_us":23673,"threads_started":1}
I20260810 14:31:54.609586 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.578632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.6.46.195:44927)
0810 14:31:54.601573 (+ 22941us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:54.601582 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:54.601613 (+ 31us) server_negotiation.cc:407] Connection header received
0810 14:31:54.601704 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:54.601711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:54.601931 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:54.602148 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:54.602724 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.604192 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.606422 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.608022 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.608179 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.608403 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:54.608923 (+ 520us) server_negotiation.cc:299] Negotiation successful
0810 14:31:54.609099 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22759,"thread_start_us":22603,"threads_started":1}
I20260810 14:31:54.611590 14024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.580344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:42453 (local address 127.0.0.1:38580)
0810 14:31:54.601856 (+ 21512us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:54.601877 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:54.601993 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:54.603449 (+ 1456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:54.603453 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:54.603482 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:54.603915 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.603923 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.605755 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.605760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.607719 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.607732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.610786 (+ 3054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.610793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.610955 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.610986 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 14:31:54.611093 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 14:31:54.611297 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21401,"thread_start_us":18167,"threads_started":1}
I20260810 14:31:54.611640 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.580894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38580 (local address 127.6.46.193:42453)
0810 14:31:54.599695 (+ 18801us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:54.599704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:54.601990 (+ 2286us) server_negotiation.cc:407] Connection header received
0810 14:31:54.602764 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:54.602771 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:54.602992 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:54.603177 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:54.604083 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.605617 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.607883 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.609338 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.609463 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.609688 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:54.611168 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0810 14:31:54.611330 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18604,"thread_start_us":18810,"threads_started":1}
I20260810 14:31:54.612891 13677 tablet_service.cc:1467] Processing CreateTablet for tablet eb9845207adf4c9b846fc6561a0de648 (DEFAULT_TABLE table=test-table [id=74c50cd230b84820843d5131b14834ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:54.613188 13904 tablet_service.cc:1467] Processing CreateTablet for tablet eb9845207adf4c9b846fc6561a0de648 (DEFAULT_TABLE table=test-table [id=74c50cd230b84820843d5131b14834ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:54.614358 13904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9845207adf4c9b846fc6561a0de648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:54.614408 13677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9845207adf4c9b846fc6561a0de648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:54.615207 13448 tablet_service.cc:1467] Processing CreateTablet for tablet eb9845207adf4c9b846fc6561a0de648 (DEFAULT_TABLE table=test-table [id=74c50cd230b84820843d5131b14834ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:54.616540 13448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb9845207adf4c9b846fc6561a0de648. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:54.626754 14032 tablet_bootstrap.cc:492] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd: Bootstrap starting.
I20260810 14:31:54.630036 14032 tablet_bootstrap.cc:654] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:54.630873 14034 tablet_bootstrap.cc:492] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb: Bootstrap starting.
I20260810 14:31:54.631538 14032 log.cc:824] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:54.633103 14035 tablet_bootstrap.cc:492] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5: Bootstrap starting.
I20260810 14:31:54.634680 14034 tablet_bootstrap.cc:654] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:54.635362 14032 tablet_bootstrap.cc:492] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd: No bootstrap required, opened a new log
I20260810 14:31:54.635622 14032 ts_tablet_manager.cc:1397] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 14:31:54.635967 14034 log.cc:824] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:54.636924 14035 tablet_bootstrap.cc:654] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:54.638080 14035 log.cc:824] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:54.639118 14034 tablet_bootstrap.cc:492] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb: No bootstrap required, opened a new log
I20260810 14:31:54.639371 14034 ts_tablet_manager.cc:1397] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 14:31:54.641026 14035 tablet_bootstrap.cc:492] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5: No bootstrap required, opened a new log
I20260810 14:31:54.641292 14035 ts_tablet_manager.cc:1397] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 14:31:54.643653 14032 raft_consensus.cc:348] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f3f4e5859d47908aa332ab9931d1fb" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38933 } } peers { permanent_uuid: "d05f51d2d5db414aa890c4ca2f01c8cd" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44927 } } peers { permanent_uuid: "289154ae263f403ca5faba66ce1c66c5" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 42453 } }
I20260810 14:31:54.644008 14032 raft_consensus.cc:374] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:54.644106 14032 raft_consensus.cc:729] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d05f51d2d5db414aa890c4ca2f01c8cd, State: Initialized, Role: FOLLOWER
I20260810 14:31:54.644665 14032 consensus_queue.cc:260] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f3f4e5859d47908aa332ab9931d1fb" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38933 } } peers { permanent_uuid: "d05f51d2d5db414aa890c4ca2f01c8cd" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44927 } } peers { permanent_uuid: "289154ae263f403ca5faba66ce1c66c5" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 42453 } }
I20260810 14:31:54.649693 14034 raft_consensus.cc:348] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f3f4e5859d47908aa332ab9931d1fb" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38933 } } peers { permanent_uuid: "d05f51d2d5db414aa890c4ca2f01c8cd" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44927 } } peers { permanent_uuid: "289154ae263f403ca5faba66ce1c66c5" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 42453 } }
I20260810 14:31:54.650169 14034 raft_consensus.cc:374] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:54.650297 14034 raft_consensus.cc:729] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f3f4e5859d47908aa332ab9931d1fb, State: Initialized, Role: FOLLOWER
I20260810 14:31:54.650743 14006 heartbeater.cc:503] Master 127.6.46.254:45391 was elected leader, sending a full tablet report...
I20260810 14:31:54.650931 14034 consensus_queue.cc:260] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f3f4e5859d47908aa332ab9931d1fb" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38933 } } peers { permanent_uuid: "d05f51d2d5db414aa890c4ca2f01c8cd" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44927 } } peers { permanent_uuid: "289154ae263f403ca5faba66ce1c66c5" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 42453 } }
I20260810 14:31:54.651301 14035 raft_consensus.cc:348] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f3f4e5859d47908aa332ab9931d1fb" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38933 } } peers { permanent_uuid: "d05f51d2d5db414aa890c4ca2f01c8cd" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44927 } } peers { permanent_uuid: "289154ae263f403ca5faba66ce1c66c5" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 42453 } }
I20260810 14:31:54.651798 14035 raft_consensus.cc:374] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:54.651938 14035 raft_consensus.cc:729] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 289154ae263f403ca5faba66ce1c66c5, State: Initialized, Role: FOLLOWER
I20260810 14:31:54.652216 14032 ts_tablet_manager.cc:1428] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd: Time spent starting tablet: real 0.016s user 0.006s sys 0.010s
I20260810 14:31:54.653210 13794 heartbeater.cc:503] Master 127.6.46.254:45391 was elected leader, sending a full tablet report...
I20260810 14:31:54.653963 14035 consensus_queue.cc:260] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f3f4e5859d47908aa332ab9931d1fb" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38933 } } peers { permanent_uuid: "d05f51d2d5db414aa890c4ca2f01c8cd" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44927 } } peers { permanent_uuid: "289154ae263f403ca5faba66ce1c66c5" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 42453 } }
I20260810 14:31:54.656136 14034 ts_tablet_manager.cc:1428] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260810 14:31:54.660223 14035 ts_tablet_manager.cc:1428] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5: Time spent starting tablet: real 0.019s user 0.013s sys 0.005s
I20260810 14:31:54.731274 14046 raft_consensus.cc:482] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:31:54.731565 14046 raft_consensus.cc:504] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f3f4e5859d47908aa332ab9931d1fb" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38933 } } peers { permanent_uuid: "d05f51d2d5db414aa890c4ca2f01c8cd" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44927 } } peers { permanent_uuid: "289154ae263f403ca5faba66ce1c66c5" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 42453 } }
I20260810 14:31:54.733295 14046 leader_election.cc:288] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02f3f4e5859d47908aa332ab9931d1fb (127.6.46.194:38933), d05f51d2d5db414aa890c4ca2f01c8cd (127.6.46.195:44927)
I20260810 14:31:54.739650 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.733878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:40503 (local address 127.6.46.195:44927)
0810 14:31:54.734091 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:54.734096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:54.734474 (+ 378us) server_negotiation.cc:407] Connection header received
0810 14:31:54.734687 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:54.734693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:54.734775 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:54.734853 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:54.735471 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.736207 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.737944 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.738744 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.738916 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.738992 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:54.739281 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 14:31:54.739406 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:31:54.739753 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.733661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:44927 (local address 127.6.46.193:40503)
0810 14:31:54.734350 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:54.734370 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:54.734515 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:54.734905 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:54.734910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:54.734937 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:54.735325 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.735335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.736335 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.736341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.737789 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.737801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.738954 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.738960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.739132 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.739157 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:31:54.739240 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 14:31:54.739368 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"spinlock_wait_cycles":36224,"thread_start_us":331,"threads_started":1}
I20260810 14:31:54.740087 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.733798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:40167 (local address 127.6.46.194:38933)
0810 14:31:54.734004 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:54.734010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:54.735084 (+ 1074us) server_negotiation.cc:407] Connection header received
0810 14:31:54.735273 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:54.735276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:54.735335 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:54.735407 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:31:54.735923 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.736621 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.738304 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.739181 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.739309 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.739381 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:54.739680 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 14:31:54.739821 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:31:54.740157 14051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.734223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:38933 (local address 127.6.46.193:40167)
0810 14:31:54.734973 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:54.734987 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:54.735137 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:54.735439 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:54.735443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:54.735466 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:31:54.735773 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.735782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.736756 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.736761 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.738158 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:54.738171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.739390 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.739397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:54.739538 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.739561 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:31:54.739639 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:31:54.739759 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":429,"threads_started":1}
I20260810 14:31:54.740893 13940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9845207adf4c9b846fc6561a0de648" candidate_uuid: "289154ae263f403ca5faba66ce1c66c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d05f51d2d5db414aa890c4ca2f01c8cd" is_pre_election: true
I20260810 14:31:54.741164 13714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9845207adf4c9b846fc6561a0de648" candidate_uuid: "289154ae263f403ca5faba66ce1c66c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f3f4e5859d47908aa332ab9931d1fb" is_pre_election: true
I20260810 14:31:54.741562 13940 raft_consensus.cc:2454] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289154ae263f403ca5faba66ce1c66c5 in term 0.
I20260810 14:31:54.741665 13714 raft_consensus.cc:2454] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 289154ae263f403ca5faba66ce1c66c5 in term 0.
I20260810 14:31:54.742271 13377 leader_election.cc:302] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 289154ae263f403ca5faba66ce1c66c5, d05f51d2d5db414aa890c4ca2f01c8cd; no voters:
I20260810 14:31:54.742660 14046 raft_consensus.cc:2781] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:31:54.742799 14046 raft_consensus.cc:482] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:31:54.742897 14046 raft_consensus.cc:3037] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:54.745005 14046 raft_consensus.cc:504] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f3f4e5859d47908aa332ab9931d1fb" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38933 } } peers { permanent_uuid: "d05f51d2d5db414aa890c4ca2f01c8cd" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44927 } } peers { permanent_uuid: "289154ae263f403ca5faba66ce1c66c5" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 42453 } }
I20260810 14:31:54.745549 14046 leader_election.cc:288] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [CANDIDATE]: Term 1 election: Requested vote from peers 02f3f4e5859d47908aa332ab9931d1fb (127.6.46.194:38933), d05f51d2d5db414aa890c4ca2f01c8cd (127.6.46.195:44927)
I20260810 14:31:54.745931 13714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9845207adf4c9b846fc6561a0de648" candidate_uuid: "289154ae263f403ca5faba66ce1c66c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02f3f4e5859d47908aa332ab9931d1fb"
W20260810 14:31:54.745999 13795 tablet.cc:2369] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:31:54.746047 13940 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb9845207adf4c9b846fc6561a0de648" candidate_uuid: "289154ae263f403ca5faba66ce1c66c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d05f51d2d5db414aa890c4ca2f01c8cd"
I20260810 14:31:54.746174 13714 raft_consensus.cc:3037] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:54.746228 13940 raft_consensus.cc:3037] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:54.749280 13714 raft_consensus.cc:2454] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289154ae263f403ca5faba66ce1c66c5 in term 1.
I20260810 14:31:54.749280 13940 raft_consensus.cc:2454] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 289154ae263f403ca5faba66ce1c66c5 in term 1.
I20260810 14:31:54.749707 13379 leader_election.cc:302] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 02f3f4e5859d47908aa332ab9931d1fb, 289154ae263f403ca5faba66ce1c66c5, d05f51d2d5db414aa890c4ca2f01c8cd; no voters:
I20260810 14:31:54.749989 14046 raft_consensus.cc:2781] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:54.750411 14046 raft_consensus.cc:686] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [term 1 LEADER]: Becoming Leader. State: Replica: 289154ae263f403ca5faba66ce1c66c5, State: Running, Role: LEADER
I20260810 14:31:54.750829 14046 consensus_queue.cc:237] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f3f4e5859d47908aa332ab9931d1fb" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38933 } } peers { permanent_uuid: "d05f51d2d5db414aa890c4ca2f01c8cd" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44927 } } peers { permanent_uuid: "289154ae263f403ca5faba66ce1c66c5" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 42453 } }
W20260810 14:31:54.754766 14008 tablet.cc:2369] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:31:54.755021 13249 catalog_manager.cc:5495] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 289154ae263f403ca5faba66ce1c66c5 (127.6.46.193). New cstate: current_term: 1 leader_uuid: "289154ae263f403ca5faba66ce1c66c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02f3f4e5859d47908aa332ab9931d1fb" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d05f51d2d5db414aa890c4ca2f01c8cd" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "289154ae263f403ca5faba66ce1c66c5" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 42453 } health_report { overall_health: HEALTHY } } }
W20260810 14:31:54.772262 13560 tablet.cc:2369] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:31:54.796339 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.791169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38590 (local address 127.6.46.193:42453)
0810 14:31:54.791350 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:54.791355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:54.791442 (+ 87us) server_negotiation.cc:407] Connection header received
0810 14:31:54.791657 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:54.791661 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:54.791736 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:54.791862 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:31:54.792237 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.792940 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.794288 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.794691 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.794813 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.795277 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:31:54.795785 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:31:54.795920 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:54.796006 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 14:31:54.796086 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:31:54.803987 13940 raft_consensus.cc:1261] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd [term 1 FOLLOWER]: Refusing update from remote peer 289154ae263f403ca5faba66ce1c66c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:54.803987 13714 raft_consensus.cc:1261] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb [term 1 FOLLOWER]: Refusing update from remote peer 289154ae263f403ca5faba66ce1c66c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:54.804909 14052 consensus_queue.cc:1035] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d05f51d2d5db414aa890c4ca2f01c8cd" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:54.805243 14046 consensus_queue.cc:1035] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02f3f4e5859d47908aa332ab9931d1fb" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:54.830358 13250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:54.830565 13250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:54.834067 13250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61f5c9666558466b9c599204c18c1256: Sending DeleteTablet for 3 replicas of tablet eb9845207adf4c9b846fc6561a0de648
I20260810 14:31:54.835100 13448 tablet_service.cc:1514] Processing DeleteTablet for tablet eb9845207adf4c9b846fc6561a0de648 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:54 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:38580
I20260810 14:31:54.835094 13677 tablet_service.cc:1514] Processing DeleteTablet for tablet eb9845207adf4c9b846fc6561a0de648 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:54 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:43200
I20260810 14:31:54.835764 13904 tablet_service.cc:1514] Processing DeleteTablet for tablet eb9845207adf4c9b846fc6561a0de648 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:54 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:51102
I20260810 14:31:54.836076 14080 tablet_replica.cc:330] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb: stopping tablet replica
I20260810 14:31:54.836459 14079 tablet_replica.cc:330] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5: stopping tablet replica
I20260810 14:31:54.836460 14080 raft_consensus.cc:2229] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:54.836872 14079 raft_consensus.cc:2229] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:31:54.836956 14080 raft_consensus.cc:2258] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:54.837479 14079 raft_consensus.cc:2258] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:54.837764 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 13351
I20260810 14:31:54.839558 14080 ts_tablet_manager.cc:1905] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:54.846841 14080 ts_tablet_manager.cc:1918] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:31:54.847105 14080 log.cc:1196] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/eb9845207adf4c9b846fc6561a0de648
I20260810 14:31:54.847409 14079 ts_tablet_manager.cc:1905] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:54.847728 14080 ts_tablet_manager.cc:1939] T eb9845207adf4c9b846fc6561a0de648 P 02f3f4e5859d47908aa332ab9931d1fb: Deleting consensus metadata
I20260810 14:31:54.849498 13225 catalog_manager.cc:4841] TS 02f3f4e5859d47908aa332ab9931d1fb (127.6.46.194:38933): tablet eb9845207adf4c9b846fc6561a0de648 (table test-table [id=74c50cd230b84820843d5131b14834ea]) successfully deleted
I20260810 14:31:54.850556 14081 tablet_replica.cc:330] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd: stopping tablet replica
I20260810 14:31:54.850960 14081 raft_consensus.cc:2229] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:54.851455 14081 raft_consensus.cc:2258] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:54.854081 14081 ts_tablet_manager.cc:1905] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:54.856814 14079 ts_tablet_manager.cc:1918] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:31:54.857035 14079 log.cc:1196] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/eb9845207adf4c9b846fc6561a0de648
I20260810 14:31:54.857662 14079 ts_tablet_manager.cc:1939] T eb9845207adf4c9b846fc6561a0de648 P 289154ae263f403ca5faba66ce1c66c5: Deleting consensus metadata
I20260810 14:31:54.859393 13223 catalog_manager.cc:4841] TS 289154ae263f403ca5faba66ce1c66c5 (127.6.46.193:42453): tablet eb9845207adf4c9b846fc6561a0de648 (table test-table [id=74c50cd230b84820843d5131b14834ea]) successfully deleted
I20260810 14:31:54.860344 14081 ts_tablet_manager.cc:1918] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:31:54.860491 14081 log.cc:1196] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/eb9845207adf4c9b846fc6561a0de648
I20260810 14:31:54.860908 14081 ts_tablet_manager.cc:1939] T eb9845207adf4c9b846fc6561a0de648 P d05f51d2d5db414aa890c4ca2f01c8cd: Deleting consensus metadata
I20260810 14:31:54.862301 13224 catalog_manager.cc:4841] TS d05f51d2d5db414aa890c4ca2f01c8cd (127.6.46.195:44927): tablet eb9845207adf4c9b846fc6561a0de648 (table test-table [id=74c50cd230b84820843d5131b14834ea]) successfully deleted
Aug 10 14:31:54 dist-test-slave-q9h9 krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:31:54.870412 14031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:54.838921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.6.46.193:42453)
0810 14:31:54.839204 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:54.839212 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:54.852263 (+ 13051us) server_negotiation.cc:407] Connection header received
0810 14:31:54.852458 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:54.852463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:54.852556 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:54.852653 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:54.854410 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.855785 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.857591 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:54.858335 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:54.858435 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:54.863587 (+ 5152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:54.863600 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:54.863613 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:54.863647 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:54.866986 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:54.868002 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:54.868011 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:54.868015 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:54.868197 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:54.868720 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:54.868743 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:54.868746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:54.869220 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:54.869502 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:54.869716 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 14:31:54.869950 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 14:31:54.871165 13552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:54.978543 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 13351
I20260810 14:31:55.005749 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 13567
Aug 10 14:31:55 dist-test-slave-q9h9 krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:31:55.020733 14025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:55.006525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43214 (local address 127.6.46.194:38933)
0810 14:31:55.006699 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:55.006706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:55.006728 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:31:55.006892 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:55.006897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:55.006993 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:55.007093 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:55.008274 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:55.009431 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:55.010740 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:55.011356 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:55.011474 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:55.015405 (+ 3931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:55.015416 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:55.015432 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:55.015471 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:55.018085 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:55.018694 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:55.018703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:55.018706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:55.018885 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:55.019239 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:55.019245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:55.019248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:55.019605 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:55.019880 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:55.020210 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 14:31:55.020415 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 14:31:55.021323 13786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:55.107980 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 13567
I20260810 14:31:55.130995 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 13799
Aug 10 14:31:55 dist-test-slave-q9h9 krb5kdc[13185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:31:55.145682 14026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:55.131761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.6.46.195:44927)
0810 14:31:55.131968 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:55.131974 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:55.131995 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:31:55.132143 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:55.132148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:55.132233 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:55.132320 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:55.133593 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:55.134392 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:55.135629 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:55.136017 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:55.136125 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:55.140336 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:55.140351 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:55.140368 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:31:55.140407 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:55.143081 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:55.143721 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:55.143728 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:55.143730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:55.143876 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:31:55.144231 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:31:55.144234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:31:55.144236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:31:55.144586 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:55.144865 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:55.145167 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 14:31:55.145390 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 14:31:55.146386 14001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:55.233211 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 13799
I20260810 14:31:55.251574 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 13201
I20260810 14:31:55.252624 13297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:55.358388 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 13201
2026-08-10T14:31:55Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4190 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T14:31:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:31:55Z Disabled control of system clock
I20260810 14:31:55.428825 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:46057
--webserver_interface=127.6.46.254
--webserver_port=0
--builtin_ntp_servers=127.6.46.212:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:46057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 14:31:55.738430 14093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:55.738686 14093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:55.738811 14093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:55.745757 14093 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:31:55.745848 14093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:55.745913 14093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:55.745986 14093 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:31:55.746065 14093 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:31:55.754858 14093 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:46057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:46057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:55.756034 14093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:55.757709 14093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:55.768581 14100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:55.768560 14103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:55.768545 14101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:55.769402 14093 server_base.cc:1034] running on GCE node
I20260810 14:31:55.770144 14093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:55.771378 14093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:55.772568 14093 hybrid_clock.cc:648] HybridClock initialized: now 1786372315772534 us; error 29 us; skew 500 ppm
I20260810 14:31:55.772872 14093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:31:55.774932 14093 webserver.cc:460] Webserver started at http://127.6.46.254:37545/ using document root <none> and password file <none>
I20260810 14:31:55.775542 14093 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:55.775633 14093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:55.775935 14093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:55.778647 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "821a05a8a0fa4edf8e4c618dd47038d5"
format_stamp: "Formatted at 2026-08-10 14:31:55 on dist-test-slave-q9h9"
server_key: "9aadf4305e60ba3ce883accd34262b29"
server_key_iv: "7cf95befbf98051ecbf03d7cb494e01b"
server_key_version: "encryptionkey@0"
I20260810 14:31:55.779379 14093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "821a05a8a0fa4edf8e4c618dd47038d5"
format_stamp: "Formatted at 2026-08-10 14:31:55 on dist-test-slave-q9h9"
server_key: "9aadf4305e60ba3ce883accd34262b29"
server_key_iv: "7cf95befbf98051ecbf03d7cb494e01b"
server_key_version: "encryptionkey@0"
I20260810 14:31:55.784027 14093 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:31:55.787153 14111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:55.788300 14093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:31:55.788450 14093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "821a05a8a0fa4edf8e4c618dd47038d5"
format_stamp: "Formatted at 2026-08-10 14:31:55 on dist-test-slave-q9h9"
server_key: "9aadf4305e60ba3ce883accd34262b29"
server_key_iv: "7cf95befbf98051ecbf03d7cb494e01b"
server_key_version: "encryptionkey@0"
I20260810 14:31:55.788637 14093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:55.793936 14093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:55.794693 14093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:55.794970 14093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:55.811066 14093 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:46057
I20260810 14:31:55.811100 14201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:46057 every 8 connection(s)
I20260810 14:31:55.812245 14093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:31:55.814049 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 14093
I20260810 14:31:55.814250 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:31:55.814868 6331 external_mini_cluster.cc:1428] Setting key b087de1a744a9016c2a986e71e0c0103
I20260810 14:31:55.815923 14203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:55.823143 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:55.816724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48116 (local address 127.6.46.254:46057)
0810 14:31:55.817772 (+ 1048us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:55.817786 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:55.817832 (+ 46us) server_negotiation.cc:407] Connection header received
0810 14:31:55.819390 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:55.819415 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:55.819922 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:55.820324 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:55.821262 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:55.821270 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:55.821286 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:55.821308 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:55.821387 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:55.821408 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:55.821633 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:55.821757 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 14:31:55.822044 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":326,"threads_started":1}
I20260810 14:31:55.828646 14203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5: Bootstrap starting.
I20260810 14:31:55.831316 14203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:55.832188 14203 log.cc:824] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:55.834797 14203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5: No bootstrap required, opened a new log
I20260810 14:31:55.841589 14203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821a05a8a0fa4edf8e4c618dd47038d5" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46057 } }
I20260810 14:31:55.841881 14203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:55.841948 14203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 821a05a8a0fa4edf8e4c618dd47038d5, State: Initialized, Role: FOLLOWER
I20260810 14:31:55.842435 14203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821a05a8a0fa4edf8e4c618dd47038d5" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46057 } }
I20260810 14:31:55.842623 14203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:31:55.842716 14203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:31:55.842837 14203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:55.844713 14203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821a05a8a0fa4edf8e4c618dd47038d5" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46057 } }
I20260810 14:31:55.845130 14203 leader_election.cc:302] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 821a05a8a0fa4edf8e4c618dd47038d5; no voters:
I20260810 14:31:55.845619 14203 leader_election.cc:288] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:31:55.845737 14209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:55.846376 14209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [term 1 LEADER]: Becoming Leader. State: Replica: 821a05a8a0fa4edf8e4c618dd47038d5, State: Running, Role: LEADER
I20260810 14:31:55.846912 14209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821a05a8a0fa4edf8e4c618dd47038d5" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46057 } }
I20260810 14:31:55.847455 14203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:31:55.849701 14212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 821a05a8a0fa4edf8e4c618dd47038d5. Latest consensus state: current_term: 1 leader_uuid: "821a05a8a0fa4edf8e4c618dd47038d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821a05a8a0fa4edf8e4c618dd47038d5" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46057 } } }
I20260810 14:31:55.850091 14212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:55.850379 14210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "821a05a8a0fa4edf8e4c618dd47038d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "821a05a8a0fa4edf8e4c618dd47038d5" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46057 } } }
I20260810 14:31:55.850739 14210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:55.852531 14221 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:31:55.856599 14221 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:31:55.864333 14221 catalog_manager.cc:1347] Generated new cluster ID: 654d64ab4c6148e2b7ba520eeddc298f
I20260810 14:31:55.864434 14221 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:31:55.881508 14221 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:31:55.881879 14221 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:31:55.886335 14221 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5: Generated new TSK 0
I20260810 14:31:55.886989 14221 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:31:55.892540 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46057
--builtin_ntp_servers=127.6.46.212:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 14:31:56.206459 14246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:56.206718 14246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:56.206848 14246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:56.213814 14246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:56.213910 14246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:56.214077 14246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:31:56.222695 14246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:56.223845 14246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:56.225643 14246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:56.237897 14254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:56.237936 14255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:56.237970 14257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:56.238806 14246 server_base.cc:1034] running on GCE node
I20260810 14:31:56.239360 14246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:56.240232 14246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:56.241463 14246 hybrid_clock.cc:648] HybridClock initialized: now 1786372316241403 us; error 46 us; skew 500 ppm
I20260810 14:31:56.241783 14246 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:31:56.243922 14246 webserver.cc:460] Webserver started at http://127.6.46.193:33077/ using document root <none> and password file <none>
I20260810 14:31:56.244602 14246 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:56.244711 14246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:56.245000 14246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:56.247879 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "a1541bc417824f5f9dd2690a98d1915b"
format_stamp: "Formatted at 2026-08-10 14:31:56 on dist-test-slave-q9h9"
server_key: "f93ba759c99dd4dd22aa176f00f484e9"
server_key_iv: "585215b745d333e5835b6e65f58c2266"
server_key_version: "encryptionkey@0"
I20260810 14:31:56.248654 14246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "a1541bc417824f5f9dd2690a98d1915b"
format_stamp: "Formatted at 2026-08-10 14:31:56 on dist-test-slave-q9h9"
server_key: "f93ba759c99dd4dd22aa176f00f484e9"
server_key_iv: "585215b745d333e5835b6e65f58c2266"
server_key_version: "encryptionkey@0"
I20260810 14:31:56.253713 14246 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 14:31:56.256971 14268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:56.258230 14246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:31:56.258368 14246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "a1541bc417824f5f9dd2690a98d1915b"
format_stamp: "Formatted at 2026-08-10 14:31:56 on dist-test-slave-q9h9"
server_key: "f93ba759c99dd4dd22aa176f00f484e9"
server_key_iv: "585215b745d333e5835b6e65f58c2266"
server_key_version: "encryptionkey@0"
I20260810 14:31:56.258576 14246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:56.263788 14246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:56.264524 14246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:56.264760 14246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:56.265741 14246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:56.267025 14246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:56.267117 14246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:56.267225 14246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:56.267294 14246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:56.295673 14246 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:33123
I20260810 14:31:56.295735 14444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:33123 every 8 connection(s)
I20260810 14:31:56.296952 14246 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:31:56.303911 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 14246
I20260810 14:31:56.304129 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:31:56.304742 6331 external_mini_cluster.cc:1428] Setting key d3118d73e3b7fef708803d452adeaec3
I20260810 14:31:56.305051 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:56.299452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:55237 (local address 127.6.46.254:46057)
0810 14:31:56.299618 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:56.299622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:56.301002 (+ 1380us) server_negotiation.cc:407] Connection header received
0810 14:31:56.302836 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:56.302841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:56.302907 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:56.302996 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:56.304348 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:56.304352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:56.304355 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:56.304372 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:56.304404 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:56.304416 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:56.304519 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:56.304714 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 14:31:56.304781 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:31:56.306092 14447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:56.299901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:46057 (local address 127.6.46.193:55237)
0810 14:31:56.300834 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:56.300870 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:56.302566 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:56.303242 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:56.303249 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:56.303339 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:56.304020 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:31:56.304022 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:56.304070 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:31:56.304168 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:56.304503 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:56.304514 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:56.304521 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:31:56.304775 (+ 254us) client_negotiation.cc:240] Negotiation successful
0810 14:31:56.305022 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":307,"threads_started":1}
I20260810 14:31:56.307607 14445 heartbeater.cc:347] Connected to a master server at 127.6.46.254:46057
I20260810 14:31:56.307842 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46057
--builtin_ntp_servers=127.6.46.212:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 14:31:56.307878 14445 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:56.308445 14445 heartbeater.cc:511] Master 127.6.46.254:46057 requested a full tablet report, sending...
I20260810 14:31:56.310647 14139 ts_manager.cc:194] Registered new tserver with Master: a1541bc417824f5f9dd2690a98d1915b (127.6.46.193:33123)
W20260810 14:31:56.624763 14448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:56.625051 14448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:56.625171 14448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:56.632611 14448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:56.632736 14448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:56.632884 14448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:31:56.641809 14448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:56.643112 14448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:56.644912 14448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:56.657189 14457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:56.657243 14456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:56.657222 14461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:56.657991 14448 server_base.cc:1034] running on GCE node
I20260810 14:31:56.658581 14448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:56.659439 14448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:56.660632 14448 hybrid_clock.cc:648] HybridClock initialized: now 1786372316660584 us; error 38 us; skew 500 ppm
I20260810 14:31:56.660909 14448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:31:56.663085 14448 webserver.cc:460] Webserver started at http://127.6.46.194:33079/ using document root <none> and password file <none>
I20260810 14:31:56.663720 14448 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:56.663823 14448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:56.664136 14448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:56.666905 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee"
format_stamp: "Formatted at 2026-08-10 14:31:56 on dist-test-slave-q9h9"
server_key: "ff6050640e047f2fe79f489084696367"
server_key_iv: "7e181ca217f0422da6d9c8d52a9c2afa"
server_key_version: "encryptionkey@0"
I20260810 14:31:56.667680 14448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee"
format_stamp: "Formatted at 2026-08-10 14:31:56 on dist-test-slave-q9h9"
server_key: "ff6050640e047f2fe79f489084696367"
server_key_iv: "7e181ca217f0422da6d9c8d52a9c2afa"
server_key_version: "encryptionkey@0"
I20260810 14:31:56.672458 14448 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:31:56.675714 14471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:56.676896 14448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:31:56.677057 14448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee"
format_stamp: "Formatted at 2026-08-10 14:31:56 on dist-test-slave-q9h9"
server_key: "ff6050640e047f2fe79f489084696367"
server_key_iv: "7e181ca217f0422da6d9c8d52a9c2afa"
server_key_version: "encryptionkey@0"
I20260810 14:31:56.677273 14448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:56.682538 14448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:56.683257 14448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:56.683494 14448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:56.684345 14448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:56.685643 14448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:56.685739 14448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:56.685842 14448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:56.685911 14448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:56.715741 14448 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:32875
I20260810 14:31:56.715801 14657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:32875 every 8 connection(s)
I20260810 14:31:56.716970 14448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:31:56.718606 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 14448
I20260810 14:31:56.718760 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:31:56.719205 6331 external_mini_cluster.cc:1428] Setting key d54a7a4e242e5505cdb562baae43494d
I20260810 14:31:56.721832 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46057
--builtin_ntp_servers=127.6.46.212:36003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 14:31:56.724798 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:56.719412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:42113 (local address 127.6.46.254:46057)
0810 14:31:56.719622 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:56.719628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:56.721133 (+ 1505us) server_negotiation.cc:407] Connection header received
0810 14:31:56.722824 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:56.722831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:56.722893 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:56.723001 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:56.724157 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:56.724162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:56.724166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:56.724183 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:56.724219 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:56.724232 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:56.724339 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:56.724479 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 14:31:56.724545 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:31:56.725575 14662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:56.719884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:46057 (local address 127.6.46.194:42113)
0810 14:31:56.720972 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:56.721026 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:56.722561 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:56.723170 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:56.723178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:56.723261 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:56.723874 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:31:56.723877 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:56.723914 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:31:56.723994 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:56.724317 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:56.724322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:56.724327 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:31:56.724524 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 14:31:56.724716 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":358,"threads_started":1}
I20260810 14:31:56.726814 14658 heartbeater.cc:347] Connected to a master server at 127.6.46.254:46057
I20260810 14:31:56.727077 14658 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:56.727613 14658 heartbeater.cc:511] Master 127.6.46.254:46057 requested a full tablet report, sending...
I20260810 14:31:56.728938 14139 ts_manager.cc:194] Registered new tserver with Master: bfed7c904bd44ab2a6fa62e7dc2307ee (127.6.46.194:32875)
W20260810 14:31:57.039964 14663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:57.040217 14663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:57.040324 14663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:57.047343 14663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:57.047439 14663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:57.047586 14663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:31:57.056537 14663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:36003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:57.057722 14663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:57.059465 14663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:57.071841 14673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:57.071962 14671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:57.072072 14669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:57.072974 14663 server_base.cc:1034] running on GCE node
I20260810 14:31:57.073510 14663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:57.074389 14663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:57.075580 14663 hybrid_clock.cc:648] HybridClock initialized: now 1786372317075535 us; error 43 us; skew 500 ppm
I20260810 14:31:57.075876 14663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:31:57.077857 14663 webserver.cc:460] Webserver started at http://127.6.46.195:37501/ using document root <none> and password file <none>
I20260810 14:31:57.078502 14663 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:57.078601 14663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:57.078902 14663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:57.081705 14663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff"
format_stamp: "Formatted at 2026-08-10 14:31:57 on dist-test-slave-q9h9"
server_key: "15ec67dc92807139796b083416f2562d"
server_key_iv: "65a2317a9688d619f909e87f13cde511"
server_key_version: "encryptionkey@0"
I20260810 14:31:57.082479 14663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff"
format_stamp: "Formatted at 2026-08-10 14:31:57 on dist-test-slave-q9h9"
server_key: "15ec67dc92807139796b083416f2562d"
server_key_iv: "65a2317a9688d619f909e87f13cde511"
server_key_version: "encryptionkey@0"
I20260810 14:31:57.087239 14663 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:31:57.090490 14687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:57.091766 14663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:31:57.091907 14663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff"
format_stamp: "Formatted at 2026-08-10 14:31:57 on dist-test-slave-q9h9"
server_key: "15ec67dc92807139796b083416f2562d"
server_key_iv: "65a2317a9688d619f909e87f13cde511"
server_key_version: "encryptionkey@0"
I20260810 14:31:57.092105 14663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:57.097353 14663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:57.098172 14663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:57.098412 14663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:57.099251 14663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:31:57.100533 14663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:31:57.100628 14663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:57.100741 14663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:31:57.100811 14663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:57.129036 14663 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:36013
I20260810 14:31:57.129097 14852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:36013 every 8 connection(s)
I20260810 14:31:57.130291 14663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:31:57.132467 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 14663
I20260810 14:31:57.132658 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:31:57.133203 6331 external_mini_cluster.cc:1428] Setting key 3fc64df6b8aa5b135341221e3cd87c07
I20260810 14:31:57.138178 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.132819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:52469 (local address 127.6.46.254:46057)
0810 14:31:57.133005 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:57.133010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:57.134734 (+ 1724us) server_negotiation.cc:407] Connection header received
0810 14:31:57.136301 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:57.136307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:57.136362 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:57.136441 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:57.137534 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:57.137538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:57.137541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:57.137557 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:57.137588 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:57.137600 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:57.137692 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:57.137869 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 14:31:57.137919 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:31:57.139074 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.133422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:46057 (local address 127.6.46.195:52469)
0810 14:31:57.134558 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:57.134611 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:57.136060 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:57.136611 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:57.136619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:57.136704 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:57.137276 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:31:57.137278 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:57.137313 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:31:57.137392 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:57.137693 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:57.137701 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:57.137708 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:31:57.137945 (+ 237us) client_negotiation.cc:240] Negotiation successful
0810 14:31:57.138210 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":367,"threads_started":1}
I20260810 14:31:57.140167 14853 heartbeater.cc:347] Connected to a master server at 127.6.46.254:46057
I20260810 14:31:57.140425 14853 heartbeater.cc:464] Registering TS with master...
I20260810 14:31:57.140973 14853 heartbeater.cc:511] Master 127.6.46.254:46057 requested a full tablet report, sending...
I20260810 14:31:57.142287 14139 ts_manager.cc:194] Registered new tserver with Master: 5bdb06cb72a14db1b7fb1465ac9bd8ff (127.6.46.195:36013)
I20260810 14:31:57.148075 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:31:57.154762 14206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.152375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.6.46.254:46057)
0810 14:31:57.152522 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:57.152527 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:57.152979 (+ 452us) server_negotiation.cc:407] Connection header received
0810 14:31:57.153161 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:57.153167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:57.153222 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:57.153306 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:57.154159 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:57.154165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:57.154170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:57.154190 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:57.154222 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:57.154233 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:57.154343 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:57.154426 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 14:31:57.154484 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 14:31:57.158316 14139 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:31:57.165052 14139 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:31:57.193019 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.176072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:32875 (local address 127.0.0.1:56034)
0810 14:31:57.188544 (+ 12472us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:57.188577 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:57.188734 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:57.190665 (+ 1931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:57.190673 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:57.190768 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:57.191687 (+ 919us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:31:57.191690 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:57.191720 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:31:57.191813 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:57.192224 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:57.192240 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:57.192248 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 14:31:57.192409 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 14:31:57.192493 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12247,"thread_start_us":15959,"threads_started":1}
I20260810 14:31:57.195405 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.176267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:36013 (local address 127.0.0.1:44896)
0810 14:31:57.192370 (+ 16103us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:57.192390 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:57.192480 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:57.193505 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:57.193510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:57.193532 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:57.194216 (+ 684us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:31:57.194219 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:57.194239 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:31:57.194292 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:57.194708 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:57.194712 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:57.194730 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 14:31:57.194858 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 14:31:57.194909 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15965,"thread_start_us":15022,"threads_started":1}
I20260810 14:31:57.195561 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.176560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.6.46.194:32875)
0810 14:31:57.186750 (+ 10190us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:57.186760 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:57.188866 (+ 2106us) server_negotiation.cc:407] Connection header received
0810 14:31:57.188944 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:57.188952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:57.189191 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:57.189337 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:57.191963 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:57.191973 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:57.191988 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:57.192014 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:57.192078 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:57.192099 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:57.194971 (+ 2872us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:57.195047 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 14:31:57.195148 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9995,"thread_start_us":9863,"threads_started":1}
I20260810 14:31:57.195861 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.176663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:33123 (local address 127.0.0.1:33626)
0810 14:31:57.192771 (+ 16108us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:57.192785 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:57.192888 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:57.194134 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:57.194137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:57.194159 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:57.194703 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:31:57.194705 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:57.194729 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:31:57.194761 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:57.195228 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:57.195232 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:57.195237 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:31:57.195358 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 14:31:57.195406 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15994,"thread_start_us":11515,"threads_started":1}
I20260810 14:31:57.196447 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.176895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33626 (local address 127.6.46.193:33123)
0810 14:31:57.188027 (+ 11132us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:57.188035 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:57.192884 (+ 4849us) server_negotiation.cc:407] Connection header received
0810 14:31:57.193000 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:57.193005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:57.193180 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:57.193311 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:57.194904 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:57.194912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:57.194924 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:57.194939 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:57.195003 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:57.195030 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:57.195162 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:57.196146 (+ 984us) server_negotiation.cc:299] Negotiation successful
0810 14:31:57.196242 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10982,"thread_start_us":13508,"threads_started":1}
I20260810 14:31:57.197181 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.177067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.6.46.195:36013)
0810 14:31:57.190508 (+ 13441us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:57.190515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:57.193049 (+ 2534us) server_negotiation.cc:407] Connection header received
0810 14:31:57.193128 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:57.193135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:57.193366 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:57.193508 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:57.194505 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:57.194513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:57.194526 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:57.194542 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:57.194600 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:57.194619 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:57.196708 (+ 2089us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:57.196795 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 14:31:57.196890 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13279,"thread_start_us":13127,"threads_started":1}
I20260810 14:31:57.199622 14346 tablet_service.cc:1467] Processing CreateTablet for tablet f2ad75b893f146dd9410f77388a9d45d (DEFAULT_TABLE table=test-table [id=b6b50eb0a7334851a714b973b54c5687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:57.199636 14558 tablet_service.cc:1467] Processing CreateTablet for tablet f2ad75b893f146dd9410f77388a9d45d (DEFAULT_TABLE table=test-table [id=b6b50eb0a7334851a714b973b54c5687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:57.200815 14346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ad75b893f146dd9410f77388a9d45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:57.200397 14756 tablet_service.cc:1467] Processing CreateTablet for tablet f2ad75b893f146dd9410f77388a9d45d (DEFAULT_TABLE table=test-table [id=b6b50eb0a7334851a714b973b54c5687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:31:57.201604 14756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ad75b893f146dd9410f77388a9d45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:57.201720 14558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ad75b893f146dd9410f77388a9d45d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:57.212543 14881 tablet_bootstrap.cc:492] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff: Bootstrap starting.
I20260810 14:31:57.214850 14882 tablet_bootstrap.cc:492] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b: Bootstrap starting.
I20260810 14:31:57.216503 14881 tablet_bootstrap.cc:654] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:57.217760 14881 log.cc:824] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:57.218406 14882 tablet_bootstrap.cc:654] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:57.219833 14882 log.cc:824] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:57.221143 14883 tablet_bootstrap.cc:492] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee: Bootstrap starting.
I20260810 14:31:57.223019 14882 tablet_bootstrap.cc:492] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b: No bootstrap required, opened a new log
I20260810 14:31:57.223297 14882 ts_tablet_manager.cc:1397] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 14:31:57.224360 14883 tablet_bootstrap.cc:654] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:57.225252 14883 log.cc:824] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:57.227461 14881 tablet_bootstrap.cc:492] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff: No bootstrap required, opened a new log
I20260810 14:31:57.227720 14881 ts_tablet_manager.cc:1397] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
I20260810 14:31:57.232453 14883 tablet_bootstrap.cc:492] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee: No bootstrap required, opened a new log
I20260810 14:31:57.232674 14883 ts_tablet_manager.cc:1397] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.007s
I20260810 14:31:57.232999 14882 raft_consensus.cc:348] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32875 } } peers { permanent_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 36013 } } peers { permanent_uuid: "a1541bc417824f5f9dd2690a98d1915b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33123 } }
I20260810 14:31:57.233412 14882 raft_consensus.cc:374] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:57.233506 14882 raft_consensus.cc:729] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1541bc417824f5f9dd2690a98d1915b, State: Initialized, Role: FOLLOWER
I20260810 14:31:57.234161 14882 consensus_queue.cc:260] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32875 } } peers { permanent_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 36013 } } peers { permanent_uuid: "a1541bc417824f5f9dd2690a98d1915b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33123 } }
I20260810 14:31:57.235275 14445 heartbeater.cc:503] Master 127.6.46.254:46057 was elected leader, sending a full tablet report...
I20260810 14:31:57.235898 14882 ts_tablet_manager.cc:1428] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260810 14:31:57.238420 14881 raft_consensus.cc:348] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32875 } } peers { permanent_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 36013 } } peers { permanent_uuid: "a1541bc417824f5f9dd2690a98d1915b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33123 } }
I20260810 14:31:57.238727 14881 raft_consensus.cc:374] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:57.238821 14881 raft_consensus.cc:729] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bdb06cb72a14db1b7fb1465ac9bd8ff, State: Initialized, Role: FOLLOWER
I20260810 14:31:57.239501 14881 consensus_queue.cc:260] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32875 } } peers { permanent_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 36013 } } peers { permanent_uuid: "a1541bc417824f5f9dd2690a98d1915b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33123 } }
I20260810 14:31:57.240536 14853 heartbeater.cc:503] Master 127.6.46.254:46057 was elected leader, sending a full tablet report...
I20260810 14:31:57.241142 14881 ts_tablet_manager.cc:1428] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260810 14:31:57.240989 14883 raft_consensus.cc:348] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32875 } } peers { permanent_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 36013 } } peers { permanent_uuid: "a1541bc417824f5f9dd2690a98d1915b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33123 } }
I20260810 14:31:57.241392 14883 raft_consensus.cc:374] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:57.241482 14883 raft_consensus.cc:729] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfed7c904bd44ab2a6fa62e7dc2307ee, State: Initialized, Role: FOLLOWER
I20260810 14:31:57.242139 14883 consensus_queue.cc:260] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32875 } } peers { permanent_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 36013 } } peers { permanent_uuid: "a1541bc417824f5f9dd2690a98d1915b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33123 } }
I20260810 14:31:57.243178 14658 heartbeater.cc:503] Master 127.6.46.254:46057 was elected leader, sending a full tablet report...
I20260810 14:31:57.243687 14883 ts_tablet_manager.cc:1428] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
W20260810 14:31:57.299168 14446 tablet.cc:2369] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:31:57.381855 14854 tablet.cc:2369] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:31:57.468963 14660 tablet.cc:2369] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:31:57.510406 14890 raft_consensus.cc:482] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:31:57.510738 14890 raft_consensus.cc:504] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32875 } } peers { permanent_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 36013 } } peers { permanent_uuid: "a1541bc417824f5f9dd2690a98d1915b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33123 } }
I20260810 14:31:57.512451 14890 leader_election.cc:288] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfed7c904bd44ab2a6fa62e7dc2307ee (127.6.46.194:32875), a1541bc417824f5f9dd2690a98d1915b (127.6.46.193:33123)
I20260810 14:31:57.514981 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.512744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:32875 (local address 127.6.46.195:58971)
0810 14:31:57.513026 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:57.513045 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:57.513157 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:57.513551 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:57.513554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:57.513575 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:57.514215 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:31:57.514217 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:57.514235 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:31:57.514272 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:57.514605 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:57.514610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:57.514616 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:31:57.514711 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 14:31:57.514765 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260810 14:31:57.515033 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.512835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:58971 (local address 127.6.46.194:32875)
0810 14:31:57.513010 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:57.513017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:57.513119 (+ 102us) server_negotiation.cc:407] Connection header received
0810 14:31:57.513338 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:57.513344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:57.513413 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:57.513506 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:57.514436 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:57.514442 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:57.514446 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:57.514464 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:57.514498 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:57.514510 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:57.514635 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:57.514745 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 14:31:57.514806 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:31:57.515565 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.513055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:59291 (local address 127.6.46.193:33123)
0810 14:31:57.513285 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:57.513292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:57.513718 (+ 426us) server_negotiation.cc:407] Connection header received
0810 14:31:57.513818 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:57.513823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:57.513876 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:57.513948 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:57.514937 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:57.514943 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:57.514947 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:57.514966 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:57.515003 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:57.515019 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:57.515131 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:57.515314 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 14:31:57.515364 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260810 14:31:57.516211 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.512886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:33123 (local address 127.6.46.195:59291)
0810 14:31:57.513606 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:57.513623 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:57.513714 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:57.514034 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:57.514039 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:57.514063 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:31:57.514709 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:31:57.514712 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:31:57.514741 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:31:57.514784 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:31:57.515172 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:31:57.515177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:31:57.515182 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:31:57.515782 (+ 600us) client_negotiation.cc:240] Negotiation successful
0810 14:31:57.515835 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":421,"threads_started":1}
I20260810 14:31:57.516037 14594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ad75b893f146dd9410f77388a9d45d" candidate_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee" is_pre_election: true
I20260810 14:31:57.516626 14594 raft_consensus.cc:2454] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bdb06cb72a14db1b7fb1465ac9bd8ff in term 0.
I20260810 14:31:57.517252 14691 leader_election.cc:302] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bdb06cb72a14db1b7fb1465ac9bd8ff, bfed7c904bd44ab2a6fa62e7dc2307ee; no voters:
I20260810 14:31:57.517221 14377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ad75b893f146dd9410f77388a9d45d" candidate_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1541bc417824f5f9dd2690a98d1915b" is_pre_election: true
I20260810 14:31:57.517603 14890 raft_consensus.cc:2781] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:31:57.517728 14890 raft_consensus.cc:482] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:31:57.517690 14377 raft_consensus.cc:2454] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5bdb06cb72a14db1b7fb1465ac9bd8ff in term 0.
I20260810 14:31:57.517822 14890 raft_consensus.cc:3037] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:57.520020 14890 raft_consensus.cc:504] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32875 } } peers { permanent_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 36013 } } peers { permanent_uuid: "a1541bc417824f5f9dd2690a98d1915b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33123 } }
I20260810 14:31:57.520628 14890 leader_election.cc:288] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [CANDIDATE]: Term 1 election: Requested vote from peers bfed7c904bd44ab2a6fa62e7dc2307ee (127.6.46.194:32875), a1541bc417824f5f9dd2690a98d1915b (127.6.46.193:33123)
I20260810 14:31:57.520992 14594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ad75b893f146dd9410f77388a9d45d" candidate_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee"
I20260810 14:31:57.521054 14377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ad75b893f146dd9410f77388a9d45d" candidate_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1541bc417824f5f9dd2690a98d1915b"
I20260810 14:31:57.521152 14594 raft_consensus.cc:3037] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:57.521240 14377 raft_consensus.cc:3037] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:57.523360 14594 raft_consensus.cc:2454] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bdb06cb72a14db1b7fb1465ac9bd8ff in term 1.
I20260810 14:31:57.523430 14377 raft_consensus.cc:2454] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5bdb06cb72a14db1b7fb1465ac9bd8ff in term 1.
I20260810 14:31:57.523693 14691 leader_election.cc:302] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bdb06cb72a14db1b7fb1465ac9bd8ff, bfed7c904bd44ab2a6fa62e7dc2307ee; no voters:
I20260810 14:31:57.523954 14890 raft_consensus.cc:2781] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:57.524358 14890 raft_consensus.cc:686] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [term 1 LEADER]: Becoming Leader. State: Replica: 5bdb06cb72a14db1b7fb1465ac9bd8ff, State: Running, Role: LEADER
I20260810 14:31:57.524844 14890 consensus_queue.cc:237] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32875 } } peers { permanent_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 36013 } } peers { permanent_uuid: "a1541bc417824f5f9dd2690a98d1915b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33123 } }
I20260810 14:31:57.529563 14139 catalog_manager.cc:5495] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 5bdb06cb72a14db1b7fb1465ac9bd8ff (127.6.46.195). New cstate: current_term: 1 leader_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5bdb06cb72a14db1b7fb1465ac9bd8ff" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 36013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a1541bc417824f5f9dd2690a98d1915b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33123 } health_report { overall_health: UNKNOWN } } }
I20260810 14:31:57.562439 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.560511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.6.46.195:36013)
0810 14:31:57.560665 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:57.560669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:57.560687 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:31:57.560917 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:57.560922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:57.561008 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:57.561100 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:57.561929 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:57.561935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:57.561941 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:57.561958 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:57.561991 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:57.562004 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:57.562152 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:57.562194 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 14:31:57.562245 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:31:57.570400 14594 raft_consensus.cc:1261] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee [term 1 FOLLOWER]: Refusing update from remote peer 5bdb06cb72a14db1b7fb1465ac9bd8ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:57.570394 14377 raft_consensus.cc:1261] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b [term 1 FOLLOWER]: Refusing update from remote peer 5bdb06cb72a14db1b7fb1465ac9bd8ff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:31:57.571310 14893 consensus_queue.cc:1035] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1541bc417824f5f9dd2690a98d1915b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:57.571640 14890 consensus_queue.cc:1035] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfed7c904bd44ab2a6fa62e7dc2307ee" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 32875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:31:57.583846 14900 mvcc.cc:205] Tried to move back new op lower bound from 7316981012758994944 to 7316981012585697280. Current Snapshot: MvccSnapshot[applied={T|T < 7316981012758994944 or (T in {7316981012758994944})}]
I20260810 14:31:57.584487 14902 mvcc.cc:205] Tried to move back new op lower bound from 7316981012758994944 to 7316981012585697280. Current Snapshot: MvccSnapshot[applied={T|T < 7316981012758994944}]
I20260810 14:31:57.595890 14139 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:57.596172 14139 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:31:57.599876 14139 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 821a05a8a0fa4edf8e4c618dd47038d5: Sending DeleteTablet for 3 replicas of tablet f2ad75b893f146dd9410f77388a9d45d
I20260810 14:31:57.600939 14756 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ad75b893f146dd9410f77388a9d45d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:57 UTC) from {username='slave'} at 127.0.0.1:44896
I20260810 14:31:57.600972 14558 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ad75b893f146dd9410f77388a9d45d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:57 UTC) from {username='slave'} at 127.0.0.1:56034
I20260810 14:31:57.601693 14346 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ad75b893f146dd9410f77388a9d45d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:31:57 UTC) from {username='slave'} at 127.0.0.1:33626
I20260810 14:31:57.601819 14916 tablet_replica.cc:330] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee: stopping tablet replica
I20260810 14:31:57.602257 14916 raft_consensus.cc:2229] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:57.602773 14916 raft_consensus.cc:2258] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:57.603833 14915 tablet_replica.cc:330] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff: stopping tablet replica
I20260810 14:31:57.604220 14915 raft_consensus.cc:2229] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:31:57.604451 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 14246
I20260810 14:31:57.604929 14915 raft_consensus.cc:2258] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:57.607280 14915 ts_tablet_manager.cc:1905] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:57.607483 14917 tablet_replica.cc:330] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b: stopping tablet replica
I20260810 14:31:57.607913 14917 raft_consensus.cc:2229] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:31:57.608435 14917 raft_consensus.cc:2258] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:31:57.608814 14916 ts_tablet_manager.cc:1905] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:57.613950 14917 ts_tablet_manager.cc:1905] T f2ad75b893f146dd9410f77388a9d45d P a1541bc417824f5f9dd2690a98d1915b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:31:57.614534 14915 ts_tablet_manager.cc:1918] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:31:57.614800 14915 log.cc:1196] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/f2ad75b893f146dd9410f77388a9d45d
I20260810 14:31:57.615293 14916 ts_tablet_manager.cc:1918] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:31:57.615429 14915 ts_tablet_manager.cc:1939] T f2ad75b893f146dd9410f77388a9d45d P 5bdb06cb72a14db1b7fb1465ac9bd8ff: Deleting consensus metadata
I20260810 14:31:57.615561 14916 log.cc:1196] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/f2ad75b893f146dd9410f77388a9d45d
I20260810 14:31:57.616088 14916 ts_tablet_manager.cc:1939] T f2ad75b893f146dd9410f77388a9d45d P bfed7c904bd44ab2a6fa62e7dc2307ee: Deleting consensus metadata
I20260810 14:31:57.617271 14113 catalog_manager.cc:4841] TS 5bdb06cb72a14db1b7fb1465ac9bd8ff (127.6.46.195:36013): tablet f2ad75b893f146dd9410f77388a9d45d (table test-table [id=b6b50eb0a7334851a714b973b54c5687]) successfully deleted
I20260810 14:31:57.617690 14117 catalog_manager.cc:4841] TS bfed7c904bd44ab2a6fa62e7dc2307ee (127.6.46.194:32875): tablet f2ad75b893f146dd9410f77388a9d45d (table test-table [id=b6b50eb0a7334851a714b973b54c5687]) successfully deleted
I20260810 14:31:57.620160 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.605928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.6.46.193:33123)
0810 14:31:57.606133 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:57.606139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:57.618190 (+ 12051us) server_negotiation.cc:407] Connection header received
0810 14:31:57.618430 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:57.618434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:57.618521 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:57.618603 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:57.619499 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:57.619505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:57.619510 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:57.619529 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:57.619558 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:57.619569 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:57.619677 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:57.619792 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 14:31:57.619857 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 14:31:57.620824 14436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:57.710101 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 14246
I20260810 14:31:57.732882 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 14448
W20260810 14:31:57.732710 14119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.46.193:33123 (error 108)
I20260810 14:31:57.734750 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.733961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:33123 (local address 127.0.0.1:33642)
0810 14:31:57.734418 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:57.734527 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:33123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":239,"threads_started":1}
W20260810 14:31:57.735097 14119 catalog_manager.cc:4584] TS a1541bc417824f5f9dd2690a98d1915b (127.6.46.193:33123): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2ad75b893f146dd9410f77388a9d45d: Network error: Client connection negotiation failed: client connection to 127.6.46.193:33123: connect: Connection refused (error 111)
I20260810 14:31:57.736490 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.734361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.6.46.194:32875)
0810 14:31:57.734530 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:57.734535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:57.734554 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:31:57.734612 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:57.734616 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:57.734741 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:57.734844 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:57.735847 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:57.735853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:57.735857 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:57.735872 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:57.735900 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:57.735912 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:57.736030 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:57.736138 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 14:31:57.736199 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:31:57.737311 14651 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:57.788977 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.788298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:33123 (local address 127.0.0.1:33646)
0810 14:31:57.788696 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:57.788809 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:33123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230}
I20260810 14:31:57.790091 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.789661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:33123 (local address 127.0.0.1:33656)
0810 14:31:57.789819 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:57.789907 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:33123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 14:31:57.824201 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 14448
I20260810 14:31:57.846134 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 14663
I20260810 14:31:57.849252 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.846961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41024 (local address 127.6.46.195:36013)
0810 14:31:57.847143 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:57.847149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:57.847168 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:31:57.847388 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:57.847394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:57.847467 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:57.847549 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:57.848421 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:57.848426 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:57.848430 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:57.848449 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:57.848486 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:57.848502 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:57.848633 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:57.848869 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 14:31:57.848943 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:31:57.850153 14848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:57.851534 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.850223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:33123 (local address 127.0.0.1:46800)
0810 14:31:57.851064 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:57.851240 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:33123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":694}
I20260810 14:31:57.852725 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.852289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:33123 (local address 127.0.0.1:46816)
0810 14:31:57.852457 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:57.852548 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:33123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 14:31:57.933562 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.933058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:33123 (local address 127.0.0.1:46830)
0810 14:31:57.933318 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:57.933422 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:33123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 14:31:57.934701 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:57.934288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:33123 (local address 127.0.0.1:46836)
0810 14:31:57.934467 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:57.934555 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:33123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260810 14:31:57.975193 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 14663
I20260810 14:31:57.999378 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 14093
I20260810 14:31:58.000514 14195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:58.103864 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 14093
2026-08-10T14:31:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2744 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 14:31:58.147791 6331 mini_oidc.cc:152] Starting JWKS server
I20260810 14:31:58.150720 6331 webserver.cc:460] Webserver started at https://127.0.0.1:35431/ using document root <none> and password file <none>
I20260810 14:31:58.151037 6331 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 14:31:58.151693 6331 webserver.cc:460] Webserver started at http://127.0.0.1:40479/ using document root <none> and password file <none>
2026-08-10T14:31:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:31:58Z Disabled control of system clock
I20260810 14:31:58.180672 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:33167
--webserver_interface=127.6.46.254
--webserver_port=0
--builtin_ntp_servers=127.6.46.212:33295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:33167
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35431/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/testchainca.cert with env {}
W20260810 14:31:58.491071 14934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:58.491361 14934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:58.491495 14934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:58.498456 14934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:31:58.498566 14934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:58.498631 14934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:58.498714 14934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:31:58.498776 14934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:31:58.498862 14934 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:31:58.498916 14934 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35431/jwks.json
I20260810 14:31:58.508343 14934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:33167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:33167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35431/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:58.509513 14934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:58.510742 14934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:58.521723 14942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:58.521790 14954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:58.521765 14941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:58.522547 14934 server_base.cc:1034] running on GCE node
I20260810 14:31:58.523219 14934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:58.524446 14934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:58.525650 14934 hybrid_clock.cc:648] HybridClock initialized: now 1786372318525603 us; error 32 us; skew 500 ppm
I20260810 14:31:58.525969 14934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:31:58.528043 14934 webserver.cc:460] Webserver started at http://127.6.46.254:42235/ using document root <none> and password file <none>
I20260810 14:31:58.528657 14934 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:58.528755 14934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:58.529040 14934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:58.531759 14934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "178182c3712648448ed035fbbf7b85c4"
format_stamp: "Formatted at 2026-08-10 14:31:58 on dist-test-slave-q9h9"
server_key: "facf5ed2a2e3ac0588d689a006ebcedb"
server_key_iv: "a8c15c4428bac75056530f6bc1c2d9ce"
server_key_version: "encryptionkey@0"
I20260810 14:31:58.532495 14934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "178182c3712648448ed035fbbf7b85c4"
format_stamp: "Formatted at 2026-08-10 14:31:58 on dist-test-slave-q9h9"
server_key: "facf5ed2a2e3ac0588d689a006ebcedb"
server_key_iv: "a8c15c4428bac75056530f6bc1c2d9ce"
server_key_version: "encryptionkey@0"
I20260810 14:31:58.537262 14934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260810 14:31:58.540431 14964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:58.541637 14934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 14:31:58.541797 14934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "178182c3712648448ed035fbbf7b85c4"
format_stamp: "Formatted at 2026-08-10 14:31:58 on dist-test-slave-q9h9"
server_key: "facf5ed2a2e3ac0588d689a006ebcedb"
server_key_iv: "a8c15c4428bac75056530f6bc1c2d9ce"
server_key_version: "encryptionkey@0"
I20260810 14:31:58.541978 14934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:58.558231 14934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:58.558914 14934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:58.559185 14934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:58.575911 14934 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:33167
I20260810 14:31:58.575929 15046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:33167 every 8 connection(s)
I20260810 14:31:58.577085 14934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:31:58.580502 15047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:58.586901 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 14934
I20260810 14:31:58.587109 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:31:58.587873 6331 external_mini_cluster.cc:1428] Setting key d0e574f888c9862fa2fca38a2cc1e4f1
I20260810 14:31:58.590883 15047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4: Bootstrap starting.
I20260810 14:31:58.593784 15047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:58.594815 15047 log.cc:824] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:58.597325 15047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4: No bootstrap required, opened a new log
I20260810 14:31:58.600749 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:58.589674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.6.46.254:33167)
0810 14:31:58.590607 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:58.590616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:58.590657 (+ 41us) server_negotiation.cc:407] Connection header received
0810 14:31:58.592340 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:58.592366 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:58.592795 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:58.593115 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:58.593737 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:58.595103 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:58.596406 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:58.597195 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:58.597374 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:58.598538 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:58.598545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:58.598558 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:58.599123 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:58.599201 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:58.599225 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:58.599332 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:58.599567 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 14:31:58.599852 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":339,"threads_started":1}
I20260810 14:31:58.604269 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:31:58.604938 15047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178182c3712648448ed035fbbf7b85c4" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33167 } }
I20260810 14:31:58.605255 15047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:58.605337 15047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178182c3712648448ed035fbbf7b85c4, State: Initialized, Role: FOLLOWER
I20260810 14:31:58.605852 15047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178182c3712648448ed035fbbf7b85c4" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33167 } }
I20260810 14:31:58.606081 15047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:31:58.606205 15047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:31:58.606359 15047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:58.608634 15047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178182c3712648448ed035fbbf7b85c4" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33167 } }
I20260810 14:31:58.609076 15047 leader_election.cc:302] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 178182c3712648448ed035fbbf7b85c4; no voters:
I20260810 14:31:58.609596 15047 leader_election.cc:288] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:31:58.609702 15058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:58.610361 15058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [term 1 LEADER]: Becoming Leader. State: Replica: 178182c3712648448ed035fbbf7b85c4, State: Running, Role: LEADER
I20260810 14:31:58.610888 15058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178182c3712648448ed035fbbf7b85c4" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33167 } }
I20260810 14:31:58.611475 15047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:31:58.613296 15061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 178182c3712648448ed035fbbf7b85c4. Latest consensus state: current_term: 1 leader_uuid: "178182c3712648448ed035fbbf7b85c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178182c3712648448ed035fbbf7b85c4" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33167 } } }
I20260810 14:31:58.613683 15061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:58.614198 15060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "178182c3712648448ed035fbbf7b85c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178182c3712648448ed035fbbf7b85c4" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33167 } } }
I20260810 14:31:58.614488 15060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4 [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:58.614650 15066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:31:58.621235 15066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:31:58.630220 15066 catalog_manager.cc:1347] Generated new cluster ID: a6e5d590340741e7ab0ec2029f4a3eb4
I20260810 14:31:58.630342 15066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:31:58.658710 15066 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:31:58.660521 15066 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:31:58.667470 15066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 178182c3712648448ed035fbbf7b85c4: Generated new TSK 0
I20260810 14:31:58.668221 15066 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 14:31:58.709681 15095 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 14:31:58.709865 15095 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 14:31:58.711766 15055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:58.689735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.6.46.254:33167)
0810 14:31:58.689911 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:58.689918 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:58.690410 (+ 492us) server_negotiation.cc:407] Connection header received
0810 14:31:58.690742 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:58.690748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:58.690834 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:58.691004 (+ 170us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:31:58.691468 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:58.692569 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:58.693884 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:58.694597 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:58.694746 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:58.695191 (+ 445us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:31:58.711161 (+ 15970us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 14:31:58.711484 (+ 323us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55144: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 14:31:58.711889 15055 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55144: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 14:31:58.713204 15094 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:31:58.689643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:33167 (local address 127.0.0.1:55144)
0810 14:31:58.690316 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:58.690332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:58.690467 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:58.691032 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:58.691035 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:58.691045 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:31:58.691331 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:58.691340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:58.692695 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:58.692698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:58.693746 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:58.693758 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:58.694746 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:58.694751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:58.694892 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:58.694947 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:31:58.711407 (+ 16460us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 14:31:58.711630 (+ 223us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.46.254:33167: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":357,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.46.254:33167: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 14:31:58.714941 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 14934
I20260810 14:31:58.715931 15042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:58.806699 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 14934
2026-08-10T14:31:58Z chronyd exiting
I20260810 14:31:58.861089 6331 test_util.cc:175] -----------------------------------------------
I20260810 14:31:58.861202 6331 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786372275129649-6331-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (718 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 14:31:58.864850 6331 mini_oidc.cc:152] Starting JWKS server
I20260810 14:31:58.866690 6331 webserver.cc:460] Webserver started at https://127.0.0.1:38703/ using document root <none> and password file <none>
I20260810 14:31:58.866921 6331 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 14:31:58.867478 6331 webserver.cc:460] Webserver started at http://127.0.0.1:38099/ using document root <none> and password file <none>
2026-08-10T14:31:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:31:58Z Disabled control of system clock
I20260810 14:31:58.897187 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43105
--webserver_interface=127.6.46.254
--webserver_port=0
--builtin_ntp_servers=127.6.46.212:44395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:43105
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38703/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/testchainca.cert with env {}
W20260810 14:31:59.209082 15105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:59.209331 15105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:59.209452 15105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:59.216481 15105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:31:59.216591 15105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:59.216653 15105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:59.216734 15105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:31:59.216792 15105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:31:59.216872 15105 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:31:59.216922 15105 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38703/jwks.json
I20260810 14:31:59.226619 15105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:43105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38703/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:59.227711 15105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:59.228940 15105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:59.239699 15111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:59.239745 15112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:59.240118 15114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:59.240396 15105 server_base.cc:1034] running on GCE node
I20260810 14:31:59.241185 15105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:59.242478 15105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:59.243706 15105 hybrid_clock.cc:648] HybridClock initialized: now 1786372319243651 us; error 41 us; skew 500 ppm
I20260810 14:31:59.244063 15105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:31:59.246158 15105 webserver.cc:460] Webserver started at http://127.6.46.254:42429/ using document root <none> and password file <none>
I20260810 14:31:59.246861 15105 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:59.246970 15105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:59.247272 15105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:31:59.249958 15105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "01eb201a9dd1449e99f5fd01297bf35b"
format_stamp: "Formatted at 2026-08-10 14:31:59 on dist-test-slave-q9h9"
server_key: "480094ed1ef39bccf5402ba563e5b99f"
server_key_iv: "af8308079fe591553cc915ea00b95a34"
server_key_version: "encryptionkey@0"
I20260810 14:31:59.250738 15105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "01eb201a9dd1449e99f5fd01297bf35b"
format_stamp: "Formatted at 2026-08-10 14:31:59 on dist-test-slave-q9h9"
server_key: "480094ed1ef39bccf5402ba563e5b99f"
server_key_iv: "af8308079fe591553cc915ea00b95a34"
server_key_version: "encryptionkey@0"
I20260810 14:31:59.255395 15105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:31:59.258589 15122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:31:59.259825 15105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:31:59.259982 15105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "01eb201a9dd1449e99f5fd01297bf35b"
format_stamp: "Formatted at 2026-08-10 14:31:59 on dist-test-slave-q9h9"
server_key: "480094ed1ef39bccf5402ba563e5b99f"
server_key_iv: "af8308079fe591553cc915ea00b95a34"
server_key_version: "encryptionkey@0"
I20260810 14:31:59.260174 15105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:31:59.280519 15105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:31:59.281246 15105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:31:59.281523 15105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:31:59.297206 15105 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:43105
I20260810 14:31:59.297228 15212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:43105 every 8 connection(s)
I20260810 14:31:59.298405 15105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:31:59.301810 15213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:31:59.303154 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 15105
I20260810 14:31:59.303349 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:31:59.303968 6331 external_mini_cluster.cc:1428] Setting key 622abec734d9b1e6df6a018f49cf93b5
I20260810 14:31:59.315160 15213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b: Bootstrap starting.
I20260810 14:31:59.317987 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:59.305723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38416 (local address 127.6.46.254:43105)
0810 14:31:59.306856 (+ 1133us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:59.306870 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:59.306920 (+ 50us) server_negotiation.cc:407] Connection header received
0810 14:31:59.308468 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:59.308500 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:59.309062 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:59.309430 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:31:59.309900 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:59.311800 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:59.313083 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:59.314083 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:59.314254 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:59.315348 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:31:59.315358 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:31:59.315375 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:31:59.316068 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:31:59.316165 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:31:59.316191 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:31:59.316308 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 14:31:59.316480 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 14:31:59.316811 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":411,"threads_started":1}
I20260810 14:31:59.318854 15213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b: Neither blocks nor log segments found. Creating new log.
I20260810 14:31:59.320106 15213 log.cc:824] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:31:59.321712 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:31:59.322808 15213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b: No bootstrap required, opened a new log
I20260810 14:31:59.329897 15213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eb201a9dd1449e99f5fd01297bf35b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43105 } }
I20260810 14:31:59.330287 15213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:31:59.330403 15213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01eb201a9dd1449e99f5fd01297bf35b, State: Initialized, Role: FOLLOWER
I20260810 14:31:59.330974 15213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eb201a9dd1449e99f5fd01297bf35b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43105 } }
I20260810 14:31:59.331207 15213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:31:59.331333 15213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:31:59.331468 15213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:31:59.333416 15213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eb201a9dd1449e99f5fd01297bf35b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43105 } }
I20260810 14:31:59.333856 15213 leader_election.cc:302] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01eb201a9dd1449e99f5fd01297bf35b; no voters:
I20260810 14:31:59.334354 15213 leader_election.cc:288] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:31:59.334465 15220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:31:59.335399 15220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [term 1 LEADER]: Becoming Leader. State: Replica: 01eb201a9dd1449e99f5fd01297bf35b, State: Running, Role: LEADER
I20260810 14:31:59.335971 15220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eb201a9dd1449e99f5fd01297bf35b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43105 } }
I20260810 14:31:59.336354 15213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:31:59.338706 15224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01eb201a9dd1449e99f5fd01297bf35b. Latest consensus state: current_term: 1 leader_uuid: "01eb201a9dd1449e99f5fd01297bf35b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eb201a9dd1449e99f5fd01297bf35b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43105 } } }
I20260810 14:31:59.338950 15223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01eb201a9dd1449e99f5fd01297bf35b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01eb201a9dd1449e99f5fd01297bf35b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43105 } } }
I20260810 14:31:59.339254 15224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:59.339265 15223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b [sys.catalog]: This master's current role is: LEADER
I20260810 14:31:59.342989 15234 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:31:59.347273 15234 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:31:59.354766 15234 catalog_manager.cc:1347] Generated new cluster ID: ede284ef16254d838b45e89e5ce2e114
I20260810 14:31:59.354878 15234 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:31:59.388839 15234 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:31:59.390074 15234 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:31:59.398105 15234 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01eb201a9dd1449e99f5fd01297bf35b: Generated new TSK 0
I20260810 14:31:59.399016 15234 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 14:31:59.451195 15260 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 14:31:59.451318 15260 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 14:31:59.452118 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:31:59.425481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38426 (local address 127.6.46.254:43105)
0810 14:31:59.425677 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 14:31:59.425683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:31:59.426050 (+ 367us) server_negotiation.cc:407] Connection header received
0810 14:31:59.426308 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:31:59.426312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:31:59.426378 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:31:59.426500 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:31:59.427015 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:59.427715 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:59.429690 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:31:59.430185 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:31:59.430284 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:59.430844 (+ 560us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:31:59.451632 (+ 20788us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 14:31:59.451900 (+ 268us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38426: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 14:31:59.452235 15216 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38426: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 14:31:59.452421 15259 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:31:59.425308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:43105 (local address 127.0.0.1:38426)
0810 14:31:59.425932 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0810 14:31:59.425950 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:31:59.426135 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:31:59.426538 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:31:59.426544 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:31:59.426557 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:31:59.426853 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:59.426865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:59.427848 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:59.427854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:59.429559 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:31:59.429575 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:31:59.430362 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:31:59.430369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:31:59.430575 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:31:59.430621 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:31:59.451815 (+ 21194us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 14:31:59.452053 (+ 238us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.46.254:43105: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38703/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":326,"threads_started":1}
I20260810 14:31:59.453660 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 15105
I20260810 14:31:59.454597 15206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:31:59.551366 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 15105
2026-08-10T14:31:59Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (746 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 14:31:59.611888 6331 mini_oidc.cc:152] Starting JWKS server
I20260810 14:31:59.613732 6331 webserver.cc:460] Webserver started at https://127.0.0.1:34107/ using document root <none> and password file <none>
I20260810 14:31:59.613948 6331 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 14:31:59.614595 6331 webserver.cc:460] Webserver started at http://127.0.0.1:45661/ using document root <none> and password file <none>
2026-08-10T14:31:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:31:59Z Disabled control of system clock
I20260810 14:31:59.645915 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:33869
--webserver_interface=127.6.46.254
--webserver_port=0
--builtin_ntp_servers=127.6.46.212:44877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:33869
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34107/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 14:31:59.958460 15269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:31:59.958710 15269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:31:59.958804 15269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:31:59.965597 15269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:31:59.965684 15269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:31:59.965727 15269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:31:59.965781 15269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:31:59.965824 15269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:31:59.965883 15269 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:31:59.965920 15269 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34107/jwks.json
I20260810 14:31:59.974494 15269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:33869
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:33869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34107/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:31:59.975603 15269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:31:59.977228 15269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:31:59.987983 15280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:59.988013 15278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:31:59.988075 15276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:31:59.988674 15269 server_base.cc:1034] running on GCE node
I20260810 14:31:59.989387 15269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:31:59.990738 15269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:31:59.991948 15269 hybrid_clock.cc:648] HybridClock initialized: now 1786372319991895 us; error 44 us; skew 500 ppm
I20260810 14:31:59.992298 15269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:31:59.998518 15269 webserver.cc:460] Webserver started at http://127.6.46.254:42291/ using document root <none> and password file <none>
I20260810 14:31:59.999169 15269 fs_manager.cc:359] Metadata directory not provided
I20260810 14:31:59.999258 15269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:31:59.999574 15269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:00.002508 15269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "04e3c5f18edd454c89bf471ab60a4738"
format_stamp: "Formatted at 2026-08-10 14:31:59 on dist-test-slave-q9h9"
server_key: "79a8c1978f025b823f1428965d0d1007"
server_key_iv: "57c152abc12d182360fd17905ae66c0a"
server_key_version: "encryptionkey@0"
I20260810 14:32:00.003242 15269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "04e3c5f18edd454c89bf471ab60a4738"
format_stamp: "Formatted at 2026-08-10 14:31:59 on dist-test-slave-q9h9"
server_key: "79a8c1978f025b823f1428965d0d1007"
server_key_iv: "57c152abc12d182360fd17905ae66c0a"
server_key_version: "encryptionkey@0"
I20260810 14:32:00.008121 15269 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 14:32:00.011355 15293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:00.012584 15269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 14:32:00.012728 15269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "04e3c5f18edd454c89bf471ab60a4738"
format_stamp: "Formatted at 2026-08-10 14:31:59 on dist-test-slave-q9h9"
server_key: "79a8c1978f025b823f1428965d0d1007"
server_key_iv: "57c152abc12d182360fd17905ae66c0a"
server_key_version: "encryptionkey@0"
I20260810 14:32:00.012909 15269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:00.031654 15269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:00.032383 15269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:00.032661 15269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:00.049135 15269 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:33869
I20260810 14:32:00.049139 15375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:33869 every 8 connection(s)
I20260810 14:32:00.050377 15269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:32:00.051826 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 15269
I20260810 14:32:00.052028 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:32:00.052879 6331 external_mini_cluster.cc:1428] Setting key 5382ebbda52871a8153e02bc77273a2d
I20260810 14:32:00.054847 15377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:00.065727 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:00.054726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58758 (local address 127.6.46.254:33869)
0810 14:32:00.055643 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:00.055654 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:00.055693 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:32:00.057231 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:00.057251 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:00.057685 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:00.057997 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:00.058571 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:00.059797 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:00.061616 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:00.062311 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:00.062447 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:00.063637 (+ 1190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:00.063644 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:00.063656 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:00.064173 (+ 517us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:00.064243 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:00.064266 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:00.064356 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:00.064591 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 14:32:00.064888 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":341,"threads_started":1}
I20260810 14:32:00.066977 15377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738: Bootstrap starting.
I20260810 14:32:00.069247 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:32:00.070052 15377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:00.070983 15377 log.cc:824] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:00.073626 15377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738: No bootstrap required, opened a new log
I20260810 14:32:00.080461 15377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e3c5f18edd454c89bf471ab60a4738" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33869 } }
I20260810 14:32:00.080783 15377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:00.080875 15377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e3c5f18edd454c89bf471ab60a4738, State: Initialized, Role: FOLLOWER
I20260810 14:32:00.081429 15377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e3c5f18edd454c89bf471ab60a4738" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33869 } }
I20260810 14:32:00.081707 15377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:32:00.081835 15377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:32:00.081985 15377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:00.083959 15377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e3c5f18edd454c89bf471ab60a4738" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33869 } }
I20260810 14:32:00.084411 15377 leader_election.cc:302] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04e3c5f18edd454c89bf471ab60a4738; no voters:
I20260810 14:32:00.084947 15377 leader_election.cc:288] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:32:00.085139 15390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:00.086140 15390 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [term 1 LEADER]: Becoming Leader. State: Replica: 04e3c5f18edd454c89bf471ab60a4738, State: Running, Role: LEADER
I20260810 14:32:00.086612 15390 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e3c5f18edd454c89bf471ab60a4738" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33869 } }
I20260810 14:32:00.086903 15377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:32:00.089278 15391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04e3c5f18edd454c89bf471ab60a4738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e3c5f18edd454c89bf471ab60a4738" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33869 } } }
I20260810 14:32:00.089695 15391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:00.090080 15392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04e3c5f18edd454c89bf471ab60a4738. Latest consensus state: current_term: 1 leader_uuid: "04e3c5f18edd454c89bf471ab60a4738" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04e3c5f18edd454c89bf471ab60a4738" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33869 } } }
I20260810 14:32:00.090409 15392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:00.091313 15402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:32:00.095618 15402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:32:00.103307 15402 catalog_manager.cc:1347] Generated new cluster ID: d3546d3069f443c79721195e0cbaeb97
I20260810 14:32:00.103458 15402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:32:00.124801 15402 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:32:00.125970 15402 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:32:00.136154 15402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04e3c5f18edd454c89bf471ab60a4738: Generated new TSK 0
I20260810 14:32:00.136904 15402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:32:00.164726 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:00.156998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58768 (local address 127.6.46.254:33869)
0810 14:32:00.157197 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:00.157207 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:00.157571 (+ 364us) server_negotiation.cc:407] Connection header received
0810 14:32:00.157764 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:00.157773 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:00.157846 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:00.157969 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:32:00.158479 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:00.159179 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:00.161120 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:00.161601 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:00.161694 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:00.162359 (+ 665us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:32:00.164104 (+ 1745us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 14:32:00.164420 (+ 316us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 14:32:00.164975 15429 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:00.156792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:33869 (local address 127.0.0.1:58768)
0810 14:32:00.157449 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:00.157469 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:00.157612 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:00.157999 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:00.158005 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:00.158039 (+ 34us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:32:00.158335 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:00.158348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:00.159305 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:00.159311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:00.160972 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:00.160987 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:00.161785 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:00.161792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:00.162000 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:00.162070 (+ 70us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:32:00.164298 (+ 2228us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 14:32:00.164599 (+ 301us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.46.254:33869: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":356,"threads_started":1}
W20260810 14:32:00.165122 15385 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58768: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260810 14:32:00.166121 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 15269
I20260810 14:32:00.167163 15368 generic_service.cc:196] Checking for leaks (request via RPC)
E20260810 14:32:00.176507 15430 webserver.cc:533] Webserver: sslize failed:
E20260810 14:32:00.176589 15430 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 14:32:00.256137 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 15269
2026-08-10T14:32:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (705 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T14:32:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:32:00Z Disabled control of system clock
I20260810 14:32:00.346002 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:38551
--webserver_interface=127.6.46.254
--webserver_port=0
--builtin_ntp_servers=127.6.46.212:45255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:38551
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/testchainca.cert with env {}
W20260810 14:32:00.654127 15438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:00.654392 15438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:00.654481 15438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:00.661347 15438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:32:00.661433 15438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:00.661476 15438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:00.661533 15438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:32:00.661576 15438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 14:32:00.661639 15438 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 14:32:00.661679 15438 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 14:32:00.670938 15438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:38551
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:38551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:00.672062 15438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:00.673207 15438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:00.683990 15450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:00.683995 15454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:00.684020 15452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:00.684736 15438 server_base.cc:1034] running on GCE node
I20260810 14:32:00.685437 15438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:00.686726 15438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:00.687915 15438 hybrid_clock.cc:648] HybridClock initialized: now 1786372320687873 us; error 33 us; skew 500 ppm
I20260810 14:32:00.688218 15438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:32:00.690215 15438 webserver.cc:460] Webserver started at http://127.6.46.254:44639/ using document root <none> and password file <none>
I20260810 14:32:00.690835 15438 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:00.690955 15438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:00.691243 15438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:00.693857 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "e91d528854d1454d8f2ba08f17a7e49c"
format_stamp: "Formatted at 2026-08-10 14:32:00 on dist-test-slave-q9h9"
server_key: "290db9b0d60d1c789ee4e3de08293587"
server_key_iv: "abef2bdf0f4f8b5d4b398762da3bbff6"
server_key_version: "encryptionkey@0"
I20260810 14:32:00.694623 15438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "e91d528854d1454d8f2ba08f17a7e49c"
format_stamp: "Formatted at 2026-08-10 14:32:00 on dist-test-slave-q9h9"
server_key: "290db9b0d60d1c789ee4e3de08293587"
server_key_iv: "abef2bdf0f4f8b5d4b398762da3bbff6"
server_key_version: "encryptionkey@0"
I20260810 14:32:00.699241 15438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:32:00.702467 15461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:00.703704 15438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:32:00.703872 15438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "e91d528854d1454d8f2ba08f17a7e49c"
format_stamp: "Formatted at 2026-08-10 14:32:00 on dist-test-slave-q9h9"
server_key: "290db9b0d60d1c789ee4e3de08293587"
server_key_iv: "abef2bdf0f4f8b5d4b398762da3bbff6"
server_key_version: "encryptionkey@0"
I20260810 14:32:00.704070 15438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:00.718628 15438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:00.719317 15438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:00.719560 15438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:00.735064 15438 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:38551
I20260810 14:32:00.735070 15553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:38551 every 8 connection(s)
I20260810 14:32:00.736239 15438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:32:00.739562 15554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:00.741925 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 15438
I20260810 14:32:00.742141 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:32:00.742753 6331 external_mini_cluster.cc:1428] Setting key 0327939afc273652b4cec9f422031fad
I20260810 14:32:00.750592 15554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c: Bootstrap starting.
I20260810 14:32:00.753863 15554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:00.755023 15554 log.cc:824] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:00.755559 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:00.744410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55826 (local address 127.6.46.254:38551)
0810 14:32:00.745243 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:00.745252 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:00.745291 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:32:00.746965 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:00.746987 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:00.747435 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:00.747762 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:00.748317 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:00.749617 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:00.751442 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:00.752090 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:00.752233 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:00.753385 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:00.753392 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:00.753405 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:00.753898 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:00.753989 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:00.754044 (+ 55us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:00.754136 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:00.754359 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 14:32:00.754636 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":291,"threads_started":1}
I20260810 14:32:00.757851 15554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c: No bootstrap required, opened a new log
I20260810 14:32:00.759315 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:32:00.766202 15554 raft_consensus.cc:348] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91d528854d1454d8f2ba08f17a7e49c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38551 } }
I20260810 14:32:00.766506 15554 raft_consensus.cc:374] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:00.766579 15554 raft_consensus.cc:729] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91d528854d1454d8f2ba08f17a7e49c, State: Initialized, Role: FOLLOWER
I20260810 14:32:00.767158 15554 consensus_queue.cc:260] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91d528854d1454d8f2ba08f17a7e49c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38551 } }
I20260810 14:32:00.767422 15554 raft_consensus.cc:388] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:32:00.767561 15554 raft_consensus.cc:482] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:32:00.767709 15554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:00.769623 15554 raft_consensus.cc:504] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91d528854d1454d8f2ba08f17a7e49c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38551 } }
I20260810 14:32:00.770056 15554 leader_election.cc:302] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e91d528854d1454d8f2ba08f17a7e49c; no voters:
I20260810 14:32:00.770557 15554 leader_election.cc:288] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:32:00.770695 15563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:00.771306 15563 raft_consensus.cc:686] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [term 1 LEADER]: Becoming Leader. State: Replica: e91d528854d1454d8f2ba08f17a7e49c, State: Running, Role: LEADER
I20260810 14:32:00.771857 15563 consensus_queue.cc:237] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91d528854d1454d8f2ba08f17a7e49c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38551 } }
I20260810 14:32:00.772449 15554 sys_catalog.cc:564] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:32:00.774765 15564 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e91d528854d1454d8f2ba08f17a7e49c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91d528854d1454d8f2ba08f17a7e49c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38551 } } }
I20260810 14:32:00.774816 15565 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e91d528854d1454d8f2ba08f17a7e49c. Latest consensus state: current_term: 1 leader_uuid: "e91d528854d1454d8f2ba08f17a7e49c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91d528854d1454d8f2ba08f17a7e49c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38551 } } }
I20260810 14:32:00.775208 15565 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:00.775660 15564 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:00.783193 15579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:32:00.788161 15579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:32:00.796787 15579 catalog_manager.cc:1347] Generated new cluster ID: 8756cbcf800c4fc28da620e800411c38
I20260810 14:32:00.796877 15579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:32:00.808455 15579 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:32:00.810150 15579 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:32:00.817924 15579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e91d528854d1454d8f2ba08f17a7e49c: Generated new TSK 0
I20260810 14:32:00.818823 15579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:32:01.318710 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:00.840036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55842 (local address 127.6.46.254:38551)
0810 14:32:00.840226 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:00.840233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:00.840683 (+ 450us) server_negotiation.cc:407] Connection header received
0810 14:32:00.840874 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:00.840881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:00.840982 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:00.841157 (+ 175us) server_negotiation.cc:226] Negotiated authn=JWT
0810 14:32:00.841558 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:00.842690 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:00.844014 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:00.844750 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:00.844902 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:00.845302 (+ 400us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 14:32:01.317885 (+472583us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 14:32:01.318404 (+ 519us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55842: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 14:32:01.318857 15558 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55842: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 14:32:01.318966 15600 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:00.839994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:38551 (local address 127.0.0.1:55842)
0810 14:32:00.840582 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:00.840597 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:00.840715 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:00.841155 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:00.841158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:00.841166 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 14:32:00.841416 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:00.841424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:00.842814 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:00.842818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:00.843869 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:00.843880 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:00.844903 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:00.844907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:00.845025 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:00.845055 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 14:32:01.318246 (+473191us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 14:32:01.318555 (+ 309us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.46.254:38551: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":315,"threads_started":1}
I20260810 14:32:01.320668 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 15438
I20260810 14:32:01.321702 15546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:01.421190 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 15438
2026-08-10T14:32:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1147 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (398 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (396 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:32:02 dist-test-slave-q9h9 krb5kdc[15618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:32:02 dist-test-slave-q9h9 krb5kdc[15618](info): set up 2 sockets
Aug 10 14:32:02 dist-test-slave-q9h9 krb5kdc[15618](info): commencing operation
krb5kdc: starting...
W20260810 14:32:03.412994 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:32:03 dist-test-slave-q9h9 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:32:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:32:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:32:03.609191 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:42279
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:42279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:03.918757 15643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:03.919018 15643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:03.919126 15643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:03.926136 15643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:32:03.926225 15643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:03.926276 15643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:03.926330 15643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:32:03.926373 15643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:32:03.935566 15643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:42279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:42279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:03.936648 15643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:03.937799 15643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:03.948613 15652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:03.948639 15655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:03.948700 15660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:03.949378 15643 server_base.cc:1034] running on GCE node
I20260810 14:32:03.950136 15643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:03.951367 15643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:03.952571 15643 hybrid_clock.cc:648] HybridClock initialized: now 1786372323952518 us; error 37 us; skew 500 ppm
I20260810 14:32:03.952888 15643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:03 dist-test-slave-q9h9 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372323, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:03.955912 15643 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:32:03.957185 15643 webserver.cc:460] Webserver started at http://127.6.46.254:41793/ using document root <none> and password file <none>
I20260810 14:32:03.957775 15643 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:03.957872 15643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:03.958175 15643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:03.960846 15643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "bc86b5d3634b484fbc4d72e0e2b51852"
format_stamp: "Formatted at 2026-08-10 14:32:03 on dist-test-slave-q9h9"
server_key: "4bc3d237c0d5418ace85b35e12fad510"
server_key_iv: "5238b5269c5e5fbfb80a28443ceb6acb"
server_key_version: "encryptionkey@0"
I20260810 14:32:03.961529 15643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "bc86b5d3634b484fbc4d72e0e2b51852"
format_stamp: "Formatted at 2026-08-10 14:32:03 on dist-test-slave-q9h9"
server_key: "4bc3d237c0d5418ace85b35e12fad510"
server_key_iv: "5238b5269c5e5fbfb80a28443ceb6acb"
server_key_version: "encryptionkey@0"
I20260810 14:32:03.966188 15643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260810 14:32:03.969340 15673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:03.970571 15643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:32:03.970719 15643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "bc86b5d3634b484fbc4d72e0e2b51852"
format_stamp: "Formatted at 2026-08-10 14:32:03 on dist-test-slave-q9h9"
server_key: "4bc3d237c0d5418ace85b35e12fad510"
server_key_iv: "5238b5269c5e5fbfb80a28443ceb6acb"
server_key_version: "encryptionkey@0"
I20260810 14:32:03.970868 15643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:04.009114 15643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:04.012763 15643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:04.012982 15643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:04.029241 15643 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:42279
I20260810 14:32:04.029280 15749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:42279 every 8 connection(s)
I20260810 14:32:04.030520 15643 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:32:04.034055 15750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:04.034960 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 15643
I20260810 14:32:04.035198 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:32:04.036042 6331 external_mini_cluster.cc:1428] Setting key 61e9f81deaff6ba0e4af997438d0ff3a
I20260810 14:32:04.046914 15750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852: Bootstrap starting.
Aug 10 14:32:04 dist-test-slave-q9h9 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:32:04.050151 15750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:04.051216 15750 log.cc:824] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:04.054050 15750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852: No bootstrap required, opened a new log
I20260810 14:32:04.057055 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:04.038351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51964 (local address 127.6.46.254:42279)
0810 14:32:04.039081 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:04.039092 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:04.039131 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:32:04.040772 (+ 1641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:04.040793 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:04.041146 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:04.041483 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:04.042879 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:04.044158 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:04.045363 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:04.046002 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:04.046198 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:04.050213 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:04.050223 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:04.050241 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:04.050284 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:04.053161 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:04.053801 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:04.053808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:04.053814 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:04.053971 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:04.054361 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:04.054365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:04.054367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:04.054966 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:04.055259 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:04.055702 (+ 443us) server_negotiation.cc:299] Negotiation successful
0810 14:32:04.056092 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":314,"threads_started":1}
I20260810 14:32:04.062656 15750 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc86b5d3634b484fbc4d72e0e2b51852" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 42279 } }
I20260810 14:32:04.062995 15750 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:04.063095 15750 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc86b5d3634b484fbc4d72e0e2b51852, State: Initialized, Role: FOLLOWER
I20260810 14:32:04.063619 15750 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc86b5d3634b484fbc4d72e0e2b51852" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 42279 } }
I20260810 14:32:04.063854 15750 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:32:04.063987 15750 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:32:04.064153 15750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:04.066707 15750 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc86b5d3634b484fbc4d72e0e2b51852" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 42279 } }
I20260810 14:32:04.067189 15750 leader_election.cc:302] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc86b5d3634b484fbc4d72e0e2b51852; no voters:
I20260810 14:32:04.067744 15750 leader_election.cc:288] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:32:04.067843 15758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:04.068508 15758 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [term 1 LEADER]: Becoming Leader. State: Replica: bc86b5d3634b484fbc4d72e0e2b51852, State: Running, Role: LEADER
I20260810 14:32:04.068998 15758 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc86b5d3634b484fbc4d72e0e2b51852" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 42279 } }
I20260810 14:32:04.069726 15750 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:32:04.071470 15759 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc86b5d3634b484fbc4d72e0e2b51852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc86b5d3634b484fbc4d72e0e2b51852" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 42279 } } }
I20260810 14:32:04.071882 15759 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:04.072548 15760 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc86b5d3634b484fbc4d72e0e2b51852. Latest consensus state: current_term: 1 leader_uuid: "bc86b5d3634b484fbc4d72e0e2b51852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc86b5d3634b484fbc4d72e0e2b51852" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 42279 } } }
I20260810 14:32:04.072926 15760 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:04.076309 15767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:32:04.081391 15767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:32:04.089474 15767 catalog_manager.cc:1347] Generated new cluster ID: 5f8847d545bc421ea5d0c3d86589c5b5
I20260810 14:32:04.089582 15767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:32:04.095794 15767 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:32:04.097523 15767 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:32:04.114606 15767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc86b5d3634b484fbc4d72e0e2b51852: Generated new TSK 0
I20260810 14:32:04.115520 15767 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:32:04.175009 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:42279
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:04.483028 15791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:04.483283 15791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:04.483410 15791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:04.490319 15791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:04.490410 15791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:04.490540 15791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:32:04.499648 15791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:04.500763 15791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:04.501962 15791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:04.513967 15798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:04.514047 15799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:04.514072 15802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:04.514894 15791 server_base.cc:1034] running on GCE node
I20260810 14:32:04.515442 15791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:04.516314 15791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:04.517510 15791 hybrid_clock.cc:648] HybridClock initialized: now 1786372324517468 us; error 31 us; skew 500 ppm
I20260810 14:32:04.517805 15791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:04 dist-test-slave-q9h9 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372324, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:04.520776 15791 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:32:04.522107 15791 webserver.cc:460] Webserver started at http://127.6.46.193:37635/ using document root <none> and password file <none>
I20260810 14:32:04.522756 15791 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:04.522874 15791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:04.523164 15791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:04.525969 15791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "a7bf3653198b452d9b8d84219f088fb5"
format_stamp: "Formatted at 2026-08-10 14:32:04 on dist-test-slave-q9h9"
server_key: "2f7c6b8f712e09c3963616625a2df47f"
server_key_iv: "b39d2cd05a488043c4be3e8c9c64c436"
server_key_version: "encryptionkey@0"
I20260810 14:32:04.526742 15791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "a7bf3653198b452d9b8d84219f088fb5"
format_stamp: "Formatted at 2026-08-10 14:32:04 on dist-test-slave-q9h9"
server_key: "2f7c6b8f712e09c3963616625a2df47f"
server_key_iv: "b39d2cd05a488043c4be3e8c9c64c436"
server_key_version: "encryptionkey@0"
I20260810 14:32:04.531414 15791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:32:04.534557 15815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:04.535737 15791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:32:04.535884 15791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "a7bf3653198b452d9b8d84219f088fb5"
format_stamp: "Formatted at 2026-08-10 14:32:04 on dist-test-slave-q9h9"
server_key: "2f7c6b8f712e09c3963616625a2df47f"
server_key_iv: "b39d2cd05a488043c4be3e8c9c64c436"
server_key_version: "encryptionkey@0"
I20260810 14:32:04.536053 15791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:04.561535 15791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:04.565148 15791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:04.565395 15791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:04.566268 15791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:04.567467 15791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:04.567539 15791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:04.567654 15791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:04.567706 15791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:04.597208 15791 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:45533
I20260810 14:32:04.597268 16019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:45533 every 8 connection(s)
I20260810 14:32:04.598428 15791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:32:04.603837 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 15791
I20260810 14:32:04.604031 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:32:04.604629 6331 external_mini_cluster.cc:1428] Setting key 055641a55b0423e9bc1c3c487007de55
Aug 10 14:32:04 dist-test-slave-q9h9 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372324, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:32:04.617427 16026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:04.600687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:42503 (local address 127.6.46.254:42279)
0810 14:32:04.601087 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:04.601092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:04.601965 (+ 873us) server_negotiation.cc:407] Connection header received
0810 14:32:04.603724 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:04.603730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:04.603807 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:04.603905 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:04.605356 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:04.606172 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:04.607621 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:04.608007 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:04.608122 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:04.611608 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:04.611614 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:04.611618 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:04.611653 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:04.614450 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:04.615269 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:04.615273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:04.615275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:04.615389 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:04.615801 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:04.615804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:04.615806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:04.616067 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:04.616205 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:04.616820 (+ 615us) server_negotiation.cc:299] Negotiation successful
0810 14:32:04.617065 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":195,"threads_started":1}
I20260810 14:32:04.618248 16027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:04.601148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:42279 (local address 127.6.46.193:42503)
0810 14:32:04.601823 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:04.601859 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:04.603461 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:04.604155 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:04.604165 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:04.604647 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:04.605192 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:04.605209 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:04.606314 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:04.606318 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:04.607477 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:04.607490 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:04.608222 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:04.608230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:04.608454 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:04.609034 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:04.609056 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:04.611382 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:04.614574 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:04.614582 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:04.614594 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:04.615136 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:04.615495 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:04.615500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:04.615504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:04.615694 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:04.616195 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:04.616203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:04.616481 (+ 278us) client_negotiation.cc:769] Sending connection context
0810 14:32:04.616717 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 14:32:04.617158 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":270,"threads_started":1}
I20260810 14:32:04.619738 16022 heartbeater.cc:347] Connected to a master server at 127.6.46.254:42279
I20260810 14:32:04.619994 16022 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:04.620638 16022 heartbeater.cc:511] Master 127.6.46.254:42279 requested a full tablet report, sending...
I20260810 14:32:04.622519 15696 ts_manager.cc:194] Registered new tserver with Master: a7bf3653198b452d9b8d84219f088fb5 (127.6.46.193:45533)
I20260810 14:32:04.623864 15696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:42503
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:32:04.653826 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:42279
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:04.977924 16031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:04.978183 16031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:04.978307 16031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:04.985140 16031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:04.985232 16031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:04.985374 16031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:32:04.994515 16031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:04.995658 16031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:04.996901 16031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:05.009281 16043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:05.009215 16040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:05.009227 16038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:05.010272 16031 server_base.cc:1034] running on GCE node
I20260810 14:32:05.010838 16031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:05.011673 16031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:05.012876 16031 hybrid_clock.cc:648] HybridClock initialized: now 1786372325012823 us; error 44 us; skew 500 ppm
I20260810 14:32:05.013154 16031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:05.018958 16031 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:32:05.020327 16031 webserver.cc:460] Webserver started at http://127.6.46.194:40515/ using document root <none> and password file <none>
I20260810 14:32:05.020957 16031 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:05.021073 16031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:05.021358 16031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:05.024127 16031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "7ba936f766214e628a733b94adb05117"
format_stamp: "Formatted at 2026-08-10 14:32:05 on dist-test-slave-q9h9"
server_key: "93aca65267817594236d487d8da9ea35"
server_key_iv: "ebce683410ff113f82eb03af49b2b987"
server_key_version: "encryptionkey@0"
I20260810 14:32:05.024878 16031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "7ba936f766214e628a733b94adb05117"
format_stamp: "Formatted at 2026-08-10 14:32:05 on dist-test-slave-q9h9"
server_key: "93aca65267817594236d487d8da9ea35"
server_key_iv: "ebce683410ff113f82eb03af49b2b987"
server_key_version: "encryptionkey@0"
I20260810 14:32:05.029790 16031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 14:32:05.033746 16052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:05.035014 16031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:32:05.035167 16031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "7ba936f766214e628a733b94adb05117"
format_stamp: "Formatted at 2026-08-10 14:32:05 on dist-test-slave-q9h9"
server_key: "93aca65267817594236d487d8da9ea35"
server_key_iv: "ebce683410ff113f82eb03af49b2b987"
server_key_version: "encryptionkey@0"
I20260810 14:32:05.035342 16031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:05.050534 16031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:05.053833 16031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:05.054090 16031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:05.054930 16031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:05.056095 16031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:05.056167 16031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:05.056254 16031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:05.056306 16031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:05.084126 16031 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:44989
I20260810 14:32:05.084159 16227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:44989 every 8 connection(s)
I20260810 14:32:05.085312 16031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:32:05.091398 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 16031
I20260810 14:32:05.091578 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:32:05.091967 6331 external_mini_cluster.cc:1428] Setting key b9868c784dab5fbe09476257a783c01f
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:32:05.104347 16026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:05.087645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:48543 (local address 127.6.46.254:42279)
0810 14:32:05.087859 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:05.087866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:05.088938 (+ 1072us) server_negotiation.cc:407] Connection header received
0810 14:32:05.090645 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:05.090649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:05.090720 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:05.090805 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:05.092238 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.093102 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.094688 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.095101 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.095221 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:05.098564 (+ 3343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:05.098570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:05.098574 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:05.098611 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:05.101533 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.102264 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.102269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.102273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.102425 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.102789 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.102792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.102795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.103086 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:05.103206 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:05.103953 (+ 747us) server_negotiation.cc:299] Negotiation successful
0810 14:32:05.104119 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 14:32:05.105616 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:05.088084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:42279 (local address 127.6.46.194:48543)
0810 14:32:05.088787 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:05.088822 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:05.090412 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:05.090983 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:05.090990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:05.091462 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:05.092064 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:05.092089 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.093249 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.093254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:05.094550 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:05.094563 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.095312 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.095320 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:05.095535 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:05.096130 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:05.096152 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:05.098355 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:05.101666 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:05.101673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:05.101683 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:05.102146 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:05.102509 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:05.102512 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:05.102514 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:05.102690 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:05.103210 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:05.103218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:05.103597 (+ 379us) client_negotiation.cc:769] Sending connection context
0810 14:32:05.103881 (+ 284us) client_negotiation.cc:240] Negotiation successful
0810 14:32:05.104378 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":274,"threads_started":1}
I20260810 14:32:05.106983 16229 heartbeater.cc:347] Connected to a master server at 127.6.46.254:42279
I20260810 14:32:05.107268 16229 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:05.107951 16229 heartbeater.cc:511] Master 127.6.46.254:42279 requested a full tablet report, sending...
I20260810 14:32:05.109364 15696 ts_manager.cc:194] Registered new tserver with Master: 7ba936f766214e628a733b94adb05117 (127.6.46.194:44989)
I20260810 14:32:05.110200 15696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:48543
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:32:05.141803 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:42279
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:44357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:05.454301 16238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:05.454536 16238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:05.454645 16238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:05.461567 16238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:05.461656 16238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:05.461792 16238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:32:05.470964 16238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:42279
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:05.472050 16238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:05.473223 16238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:05.485513 16248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:05.485582 16253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:05.485736 16255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:05.486531 16238 server_base.cc:1034] running on GCE node
I20260810 14:32:05.487057 16238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:05.487881 16238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:05.489071 16238 hybrid_clock.cc:648] HybridClock initialized: now 1786372325489019 us; error 43 us; skew 500 ppm
I20260810 14:32:05.489354 16238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:05.492321 16238 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:32:05.493597 16238 webserver.cc:460] Webserver started at http://127.6.46.195:37173/ using document root <none> and password file <none>
I20260810 14:32:05.494241 16238 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:05.494355 16238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:05.494632 16238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:05.497308 16238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "1437dfabd6324feab153636424a01291"
format_stamp: "Formatted at 2026-08-10 14:32:05 on dist-test-slave-q9h9"
server_key: "b5814b8e5e097333e4b85a04a13bf56d"
server_key_iv: "26574435946d7c3d9bde75d21f04ce33"
server_key_version: "encryptionkey@0"
I20260810 14:32:05.498036 16238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "1437dfabd6324feab153636424a01291"
format_stamp: "Formatted at 2026-08-10 14:32:05 on dist-test-slave-q9h9"
server_key: "b5814b8e5e097333e4b85a04a13bf56d"
server_key_iv: "26574435946d7c3d9bde75d21f04ce33"
server_key_version: "encryptionkey@0"
I20260810 14:32:05.502647 16238 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:32:05.505760 16265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:05.506924 16238 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:32:05.507063 16238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "1437dfabd6324feab153636424a01291"
format_stamp: "Formatted at 2026-08-10 14:32:05 on dist-test-slave-q9h9"
server_key: "b5814b8e5e097333e4b85a04a13bf56d"
server_key_iv: "26574435946d7c3d9bde75d21f04ce33"
server_key_version: "encryptionkey@0"
I20260810 14:32:05.507244 16238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:05.531179 16238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:05.534894 16238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:05.535142 16238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:05.535956 16238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:05.537164 16238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:05.537240 16238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:05.537351 16238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:05.537405 16238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:05.566991 16238 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:41747
I20260810 14:32:05.567070 16459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:41747 every 8 connection(s)
I20260810 14:32:05.568150 16238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:32:05.570906 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 16238
I20260810 14:32:05.571115 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:32:05.571668 6331 external_mini_cluster.cc:1428] Setting key 9fab61a474235919ce92702e8b11df47
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:32:05.586455 16026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:05.570514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:53121 (local address 127.6.46.254:42279)
0810 14:32:05.570682 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:05.570687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:05.572009 (+ 1322us) server_negotiation.cc:407] Connection header received
0810 14:32:05.574250 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:05.574254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:05.574321 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:05.574398 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:05.575860 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.576589 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.577972 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.578422 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.578516 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:05.581516 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:05.581520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:05.581524 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:05.581557 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:05.583788 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.584404 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.584408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.584410 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.584544 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.584867 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.584870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.584872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.585155 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:05.585302 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:05.585939 (+ 637us) server_negotiation.cc:299] Negotiation successful
0810 14:32:05.586171 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:32:05.587417 16466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:05.571072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:42279 (local address 127.6.46.195:53121)
0810 14:32:05.571863 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:05.571907 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:05.573968 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:05.574643 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:05.574654 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:05.575199 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:05.575719 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:05.575736 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.576713 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.576717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:05.577870 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:05.577881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.578596 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.578601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:05.578799 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:05.579342 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:05.579362 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:05.581322 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:05.583903 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:05.583909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:05.583919 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:05.584287 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:05.584629 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:05.584632 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:05.584634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:05.584792 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:05.585299 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:05.585305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:05.585598 (+ 293us) client_negotiation.cc:769] Sending connection context
0810 14:32:05.585877 (+ 279us) client_negotiation.cc:240] Negotiation successful
0810 14:32:05.586327 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":322,"threads_started":1}
I20260810 14:32:05.588538 16461 heartbeater.cc:347] Connected to a master server at 127.6.46.254:42279
I20260810 14:32:05.588797 16461 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:05.589434 16461 heartbeater.cc:511] Master 127.6.46.254:42279 requested a full tablet report, sending...
I20260810 14:32:05.590728 15696 ts_manager.cc:194] Registered new tserver with Master: 1437dfabd6324feab153636424a01291 (127.6.46.195:41747)
I20260810 14:32:05.591482 15696 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:53121
I20260810 14:32:05.598896 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/test-admin.keytab.
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:05.626670 16022 heartbeater.cc:503] Master 127.6.46.254:42279 was elected leader, sending a full tablet report...
I20260810 14:32:05.627374 6331 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 14:32:05.635051 16476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:05.632630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.6.46.193:45533)
0810 14:32:05.633270 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:05.633277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:05.633303 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:32:05.633362 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:05.633368 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:05.633564 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:05.633718 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:05.634582 (+ 864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38848: BlockingRecv error: recv got EOF from 127.0.0.1:38848 (error 108)
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":337,"threads_started":1}
W20260810 14:32:05.635200 6331 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/krb5cc' of type FILE
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:05.636885 6331 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:05.638193 6331 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:32:05.652299 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:05.639733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38852 (local address 127.6.46.193:45533)
0810 14:32:05.639906 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:05.639912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:05.640186 (+ 274us) server_negotiation.cc:407] Connection header received
0810 14:32:05.640307 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:05.640310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:05.640361 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:05.640434 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:05.641483 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.642507 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.643673 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.644196 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.644295 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:05.647479 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:05.647489 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:05.647502 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:05.647540 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:05.650004 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.650582 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.650588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.650590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.650720 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.650997 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.651000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.651002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.651331 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:05.651546 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:05.651803 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 14:32:05.652028 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 14:32:05.657178 16476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:05.655815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.6.46.193:45533)
0810 14:32:05.656003 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:05.656009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:05.656214 (+ 205us) server_negotiation.cc:407] Connection header received
0810 14:32:05.656336 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:05.656339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:05.656389 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:05.656452 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:05.656954 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38868: BlockingRecv error: recv got EOF from 127.0.0.1:38868 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 14:32:05.657332 6331 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/krb5cc' of type FILE
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:05.658779 6331 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:05.659986 6331 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:32:05.672082 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:05.661116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.6.46.193:45533)
0810 14:32:05.661281 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:05.661286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:05.661449 (+ 163us) server_negotiation.cc:407] Connection header received
0810 14:32:05.661563 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:05.661566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:05.661613 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:05.661677 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:05.662557 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.663198 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.664268 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.664662 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.664742 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:05.667771 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:05.667776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:05.667779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:05.667807 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:05.670118 (+ 2311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.670686 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.670689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.670692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.670807 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.671092 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.671095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.671097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.671329 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:05.671511 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:05.671683 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 14:32:05.671818 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 14:32:05.675675 16476 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:05.674302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.6.46.193:45533)
0810 14:32:05.674461 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:05.674466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:05.674652 (+ 186us) server_negotiation.cc:407] Connection header received
0810 14:32:05.674755 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:05.674758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:05.674837 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:05.674902 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:05.675498 (+ 596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38894: BlockingRecv error: recv got EOF from 127.0.0.1:38894 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 14:32:05.675897 6331 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786372275129649-6331-0/krb5kdc/krb5cc' of type FILE
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:05.677210 6331 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:05.678413 6331 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:32:05.690248 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:05.679485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38898 (local address 127.6.46.193:45533)
0810 14:32:05.679662 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:05.679667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:05.679844 (+ 177us) server_negotiation.cc:407] Connection header received
0810 14:32:05.679941 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:05.679945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:05.680006 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:05.680102 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:05.680875 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.681525 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.682659 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.683054 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.683188 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:05.686133 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:05.686138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:05.686141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:05.686168 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:05.688271 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.688812 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.688815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.688817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.688928 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.689187 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.689190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.689192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.689434 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:05.689642 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:05.689786 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 14:32:05.689939 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:32:05.691465 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 15791
I20260810 14:32:05.701674 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:05.692143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.6.46.193:45533)
0810 14:32:05.692310 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:05.692315 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:05.692456 (+ 141us) server_negotiation.cc:407] Connection header received
0810 14:32:05.692585 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:05.692587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:05.692633 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:05.692703 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:05.693480 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.694127 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.695226 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.695627 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.695720 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:05.697339 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:05.697343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:05.697346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:05.697372 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:05.699560 (+ 2188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.700104 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.700107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.700109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.700216 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.700601 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.700604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.700606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.700830 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:05.701007 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:05.701251 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 14:32:05.701385 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:32:05.702302 16014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:05.783830 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 15791
I20260810 14:32:05.807288 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 16031
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:32:05.821777 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:05.808162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.6.46.194:44989)
0810 14:32:05.808679 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:05.808687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:05.808713 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:32:05.808773 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:05.808779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:05.808984 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:05.809114 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:05.809999 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.811170 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.812470 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.813010 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.813116 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:05.816705 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:05.816716 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:05.816730 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:05.816760 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:05.819407 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.819980 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.819986 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.819989 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.820100 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.820368 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.820371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.820373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.820694 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:05.820922 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:05.821221 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 14:32:05.821487 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":251,"threads_started":1}
I20260810 14:32:05.823103 16219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:05.904672 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 16031
I20260810 14:32:05.928099 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 16238
Aug 10 14:32:05 dist-test-slave-q9h9 krb5kdc[15618](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:32:05.942698 16506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:05.929064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48608 (local address 127.6.46.195:41747)
0810 14:32:05.929648 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:05.929655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:05.929682 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:32:05.929744 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:05.929751 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:05.929956 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:05.930120 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:05.931046 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.932075 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.933222 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:05.933758 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:05.933850 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:05.937594 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:05.937605 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:05.937619 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:05.937655 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:05.940290 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.940821 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.940827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.940829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.940947 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:05.941263 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:05.941268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:05.941270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:05.941628 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:05.941861 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:05.942111 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 14:32:05.942325 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":275,"threads_started":1}
I20260810 14:32:05.943988 16453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:06.024807 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 16238
I20260810 14:32:06.044497 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 15643
I20260810 14:32:06.045663 15739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:06.146852 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 15643
2026-08-10T14:32:06Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3934 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:32:06 dist-test-slave-q9h9 krb5kdc[16515](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:32:06 dist-test-slave-q9h9 krb5kdc[16515](info): set up 2 sockets
Aug 10 14:32:06 dist-test-slave-q9h9 krb5kdc[16515](info): commencing operation
krb5kdc: starting...
W20260810 14:32:07.367461 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.171s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:32:07 dist-test-slave-q9h9 krb5kdc[16515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:32:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:32:07Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.254.keytab.
Entry for principal oryx/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.254.keytab.
I20260810 14:32:07.565012 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:39189
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.254.keytab
--principal=oryx/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:39189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:07.876659 16536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:07.876936 16536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:07.877074 16536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:07.886904 16536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:32:07.887112 16536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:07.887221 16536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:07.887324 16536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:32:07.887403 16536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:32:07.898245 16536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:39189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.254.keytab
--principal=oryx/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:39189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:07.899418 16536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:07.900620 16536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:07.912577 16545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:07.912514 16548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:07.912523 16546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:07.913321 16536 server_base.cc:1034] running on GCE node
I20260810 14:32:07.914039 16536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:07.915287 16536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:07.916529 16536 hybrid_clock.cc:648] HybridClock initialized: now 1786372327916477 us; error 29 us; skew 500 ppm
I20260810 14:32:07.916868 16536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:07 dist-test-slave-q9h9 krb5kdc[16515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372327, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:07.919795 16536 init.cc:377] Logged in from keytab as oryx/127.6.46.254@KRBTEST.COM (short username oryx)
I20260810 14:32:07.921190 16536 webserver.cc:460] Webserver started at http://127.6.46.254:38219/ using document root <none> and password file <none>
I20260810 14:32:07.921803 16536 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:07.921916 16536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:07.922243 16536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:07.925010 16536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "bd97ad2506a34450b6cf70414423528c"
format_stamp: "Formatted at 2026-08-10 14:32:07 on dist-test-slave-q9h9"
server_key: "9b142b92a69e4a08ee43dfd4f0dd36be"
server_key_iv: "c149ea48bd5c258bf395ddbd67b2e813"
server_key_version: "encryptionkey@0"
I20260810 14:32:07.925740 16536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "bd97ad2506a34450b6cf70414423528c"
format_stamp: "Formatted at 2026-08-10 14:32:07 on dist-test-slave-q9h9"
server_key: "9b142b92a69e4a08ee43dfd4f0dd36be"
server_key_iv: "c149ea48bd5c258bf395ddbd67b2e813"
server_key_version: "encryptionkey@0"
I20260810 14:32:07.930491 16536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 14:32:07.933571 16556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:07.934820 16536 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:32:07.934971 16536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "bd97ad2506a34450b6cf70414423528c"
format_stamp: "Formatted at 2026-08-10 14:32:07 on dist-test-slave-q9h9"
server_key: "9b142b92a69e4a08ee43dfd4f0dd36be"
server_key_iv: "c149ea48bd5c258bf395ddbd67b2e813"
server_key_version: "encryptionkey@0"
I20260810 14:32:07.935154 16536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:07.952855 16536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:07.956378 16536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:07.956619 16536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:07.972965 16536 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:39189
I20260810 14:32:07.973001 16645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:39189 every 8 connection(s)
I20260810 14:32:07.974154 16536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:32:07.977504 16647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:07.981245 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 16536
I20260810 14:32:07.981472 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:32:07.982290 6331 external_mini_cluster.cc:1428] Setting key b13e01b88cb46022c469f5fedaf71c94
I20260810 14:32:07.988480 16647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c: Bootstrap starting.
I20260810 14:32:07.991801 16647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:07.993021 16647 log.cc:824] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:07.995635 16647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c: No bootstrap required, opened a new log
Aug 10 14:32:07 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
I20260810 14:32:08.004333 16647 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } }
I20260810 14:32:08.004673 16647 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:08.004782 16647 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd97ad2506a34450b6cf70414423528c, State: Initialized, Role: FOLLOWER
I20260810 14:32:08.005272 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:07.984221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.6.46.254:39189)
0810 14:32:07.984935 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:07.984945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:07.984984 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:32:07.987103 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:07.987124 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:07.987449 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:07.987760 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:07.989494 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:07.990821 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:07.992484 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:07.993441 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:07.993633 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:07.997812 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:07.997823 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:07.997836 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:07.997887 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:08.001135 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:08.001931 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:08.001937 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:08.001943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:08.002180 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:08.002577 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:08.002580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:08.002582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:08.003156 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:08.003429 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:08.003790 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 14:32:08.004118 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":336,"threads_started":1}
I20260810 14:32:08.005313 16647 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } }
I20260810 14:32:08.005544 16647 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:32:08.005676 16647 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:32:08.005861 16647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:08.008081 16647 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } }
I20260810 14:32:08.008517 16647 leader_election.cc:302] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd97ad2506a34450b6cf70414423528c; no voters:
I20260810 14:32:08.009087 16647 leader_election.cc:288] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:32:08.009224 16655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:08.010310 16655 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [term 1 LEADER]: Becoming Leader. State: Replica: bd97ad2506a34450b6cf70414423528c, State: Running, Role: LEADER
I20260810 14:32:08.010952 16655 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } }
I20260810 14:32:08.010950 16647 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:32:08.013845 16658 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd97ad2506a34450b6cf70414423528c. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } }
I20260810 14:32:08.013895 16657 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } }
I20260810 14:32:08.014272 16658 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:08.014334 16657 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:08.020166 16665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:32:08.024818 16665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:32:08.033869 16665 catalog_manager.cc:1347] Generated new cluster ID: dfcf555871a444528a1ebb2de1eda106
I20260810 14:32:08.033962 16665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:32:08.050145 16665 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:32:08.051247 16665 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:32:08.066457 16665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c: Generated new TSK 0
I20260810 14:32:08.067225 16665 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab.
Entry for principal oryx/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab.
I20260810 14:32:08.136659 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:39189
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab
--principal=oryx/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:08.469048 16693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:08.469301 16693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:08.469388 16693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:08.476303 16693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:08.476392 16693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:08.476501 16693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:32:08.485487 16693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab
--principal=oryx/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:39189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:08.486634 16693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:08.487807 16693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:08.499667 16702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:08.499732 16714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:08.499760 16707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:08.500612 16693 server_base.cc:1034] running on GCE node
I20260810 14:32:08.501114 16693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:08.501896 16693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:08.503088 16693 hybrid_clock.cc:648] HybridClock initialized: now 1786372328503035 us; error 51 us; skew 500 ppm
I20260810 14:32:08.503342 16693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:08 dist-test-slave-q9h9 krb5kdc[16515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372328, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:08.506412 16693 init.cc:377] Logged in from keytab as oryx/127.6.46.193@KRBTEST.COM (short username oryx)
I20260810 14:32:08.507707 16693 webserver.cc:460] Webserver started at http://127.6.46.193:33861/ using document root <none> and password file <none>
I20260810 14:32:08.508308 16693 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:08.508401 16693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:08.508632 16693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:08.511293 16693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "a6e4c148a686452caad3c9f8cb9bf56c"
format_stamp: "Formatted at 2026-08-10 14:32:08 on dist-test-slave-q9h9"
server_key: "7d5e9f329f3bb6c8157cf7d35ad6dbe5"
server_key_iv: "fe52127590f497ef02f0d2ea22b7822b"
server_key_version: "encryptionkey@0"
I20260810 14:32:08.512063 16693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "a6e4c148a686452caad3c9f8cb9bf56c"
format_stamp: "Formatted at 2026-08-10 14:32:08 on dist-test-slave-q9h9"
server_key: "7d5e9f329f3bb6c8157cf7d35ad6dbe5"
server_key_iv: "fe52127590f497ef02f0d2ea22b7822b"
server_key_version: "encryptionkey@0"
I20260810 14:32:08.516929 16693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260810 14:32:08.520152 16723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:08.521324 16693 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:32:08.521454 16693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "a6e4c148a686452caad3c9f8cb9bf56c"
format_stamp: "Formatted at 2026-08-10 14:32:08 on dist-test-slave-q9h9"
server_key: "7d5e9f329f3bb6c8157cf7d35ad6dbe5"
server_key_iv: "fe52127590f497ef02f0d2ea22b7822b"
server_key_version: "encryptionkey@0"
I20260810 14:32:08.521607 16693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:08.531714 16693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:08.535207 16693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:08.535425 16693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:08.536217 16693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:08.537384 16693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:08.537454 16693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:08.537539 16693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:08.537581 16693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:08.567538 16693 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:36107
I20260810 14:32:08.567595 16901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:36107 every 8 connection(s)
I20260810 14:32:08.568763 16693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:32:08.577697 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 16693
I20260810 14:32:08.577914 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:32:08.578562 6331 external_mini_cluster.cc:1428] Setting key 5774b518b5119ce23f56ddf970fcf1cf
Aug 10 14:32:08 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372328, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
I20260810 14:32:08.588104 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:08.571293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:42929 (local address 127.6.46.254:39189)
0810 14:32:08.571742 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:08.571746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:08.572581 (+ 835us) server_negotiation.cc:407] Connection header received
0810 14:32:08.574373 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:08.574380 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:08.574459 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:08.574555 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:08.576022 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:08.576748 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:08.578355 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:08.578776 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:08.578899 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:08.582718 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:08.582722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:08.582726 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:08.582757 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:08.585106 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:08.585901 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:08.585904 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:08.585906 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:08.586041 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:08.586500 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:08.586502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:08.586504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:08.586763 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:08.586882 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:08.587689 (+ 807us) server_negotiation.cc:299] Negotiation successful
0810 14:32:08.587856 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":249,"threads_started":1}
I20260810 14:32:08.589375 16906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:08.571722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.193:42929)
0810 14:32:08.572418 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:08.572466 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:08.574110 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:08.574809 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:08.574817 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:08.575308 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:08.575847 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:08.575864 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:08.576894 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:08.576898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:08.578209 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:08.578223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:08.578991 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:08.578999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:08.579232 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:08.579928 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:08.579950 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:08.582493 (+ 2543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:08.585240 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:08.585250 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:08.585265 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:08.585802 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:08.586139 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:08.586143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:08.586146 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:08.586406 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:08.586874 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:08.586881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:08.587301 (+ 420us) client_negotiation.cc:769] Sending connection context
0810 14:32:08.587579 (+ 278us) client_negotiation.cc:240] Negotiation successful
0810 14:32:08.588083 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":270,"threads_started":1}
I20260810 14:32:08.591041 16902 heartbeater.cc:347] Connected to a master server at 127.6.46.254:39189
I20260810 14:32:08.591361 16902 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:08.592031 16902 heartbeater.cc:511] Master 127.6.46.254:39189 requested a full tablet report, sending...
I20260810 14:32:08.594271 16580 ts_manager.cc:194] Registered new tserver with Master: a6e4c148a686452caad3c9f8cb9bf56c (127.6.46.193:36107)
I20260810 14:32:08.596200 16580 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:42929
WARNING: no policy specified for oryx/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab.
Entry for principal oryx/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab.
I20260810 14:32:08.630998 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:39189
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab
--principal=oryx/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:08.945338 16910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:08.945569 16910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:08.945691 16910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:08.952565 16910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:08.952656 16910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:08.952800 16910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:32:08.961882 16910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab
--principal=oryx/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:39189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:08.963248 16910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:08.964500 16910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:08.976660 16922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:08.976755 16925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:08.977152 16921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:08.977458 16910 server_base.cc:1034] running on GCE node
I20260810 14:32:08.978096 16910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:08.979071 16910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:08.980285 16910 hybrid_clock.cc:648] HybridClock initialized: now 1786372328980224 us; error 49 us; skew 500 ppm
I20260810 14:32:08.980649 16910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:08 dist-test-slave-q9h9 krb5kdc[16515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372328, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:08.984200 16910 init.cc:377] Logged in from keytab as oryx/127.6.46.194@KRBTEST.COM (short username oryx)
I20260810 14:32:08.985721 16910 webserver.cc:460] Webserver started at http://127.6.46.194:45173/ using document root <none> and password file <none>
I20260810 14:32:08.986538 16910 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:08.986671 16910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:08.986972 16910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:08.990558 16910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "7358de53372848ca93f3d28c6e674f2d"
format_stamp: "Formatted at 2026-08-10 14:32:08 on dist-test-slave-q9h9"
server_key: "18dbb1b5d588d6bf3d742c350968833d"
server_key_iv: "da7fa3022036aa232e2b8453e1b7f519"
server_key_version: "encryptionkey@0"
I20260810 14:32:08.991433 16910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "7358de53372848ca93f3d28c6e674f2d"
format_stamp: "Formatted at 2026-08-10 14:32:08 on dist-test-slave-q9h9"
server_key: "18dbb1b5d588d6bf3d742c350968833d"
server_key_iv: "da7fa3022036aa232e2b8453e1b7f519"
server_key_version: "encryptionkey@0"
I20260810 14:32:08.996197 16910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:32:08.999341 16932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:09.000583 16910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 14:32:09.000710 16910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "7358de53372848ca93f3d28c6e674f2d"
format_stamp: "Formatted at 2026-08-10 14:32:08 on dist-test-slave-q9h9"
server_key: "18dbb1b5d588d6bf3d742c350968833d"
server_key_iv: "da7fa3022036aa232e2b8453e1b7f519"
server_key_version: "encryptionkey@0"
I20260810 14:32:09.000866 16910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:09.012624 16910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:09.016124 16910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:09.016350 16910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:09.017140 16910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:09.018354 16910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:09.018425 16910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:09.018512 16910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:09.018555 16910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:09.046306 16910 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:38723
I20260810 14:32:09.046351 17137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:38723 every 8 connection(s)
I20260810 14:32:09.047387 16910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:32:09.051123 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 16910
I20260810 14:32:09.051296 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:32:09.051718 6331 external_mini_cluster.cc:1428] Setting key 32f19b9fffa2fc95175e061f2342a917
Aug 10 14:32:09 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372328, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
I20260810 14:32:09.066329 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:09.049623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:44115 (local address 127.6.46.254:39189)
0810 14:32:09.049790 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:09.049794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:09.050872 (+ 1078us) server_negotiation.cc:407] Connection header received
0810 14:32:09.052752 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:09.052758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:09.052837 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:09.052923 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:09.054470 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:09.055189 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:09.056669 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:09.057080 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:09.057191 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:09.060524 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:09.060529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:09.060532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:09.060562 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:09.063529 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:09.064174 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:09.064177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:09.064180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:09.064302 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:09.064712 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:09.064716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:09.064718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:09.064982 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:09.065123 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:09.065763 (+ 640us) server_negotiation.cc:299] Negotiation successful
0810 14:32:09.065981 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:32:09.067167 17142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:09.050054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.194:44115)
0810 14:32:09.050726 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:09.050762 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:09.052502 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:09.053146 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:09.053155 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:09.053746 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:09.054309 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:09.054327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:09.055344 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:09.055348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:09.056517 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:09.056530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:09.057292 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:09.057300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:09.057518 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:09.058128 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:09.058149 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:09.060313 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:09.063640 (+ 3327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:09.063649 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:09.063662 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:09.064051 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:09.064406 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:09.064411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:09.064415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:09.064587 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:09.065109 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:09.065116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:09.065405 (+ 289us) client_negotiation.cc:769] Sending connection context
0810 14:32:09.065621 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 14:32:09.066106 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":249,"threads_started":1}
I20260810 14:32:09.068905 17138 heartbeater.cc:347] Connected to a master server at 127.6.46.254:39189
I20260810 14:32:09.069177 17138 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:09.069821 17138 heartbeater.cc:511] Master 127.6.46.254:39189 requested a full tablet report, sending...
I20260810 14:32:09.071229 16580 ts_manager.cc:194] Registered new tserver with Master: 7358de53372848ca93f3d28c6e674f2d (127.6.46.194:38723)
I20260810 14:32:09.071952 16580 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:44115
WARNING: no policy specified for oryx/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab.
Entry for principal oryx/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab.
I20260810 14:32:09.098529 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:39189
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab
--principal=oryx/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:09.420001 17148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:09.420275 17148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:09.420384 17148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:09.427702 17148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:09.427834 17148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:09.428001 17148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:32:09.437649 17148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab
--principal=oryx/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:39189
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:09.438970 17148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:09.440371 17148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:09.453190 17155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:09.453266 17156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:09.453523 17161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:09.454273 17148 server_base.cc:1034] running on GCE node
I20260810 14:32:09.454877 17148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:09.455763 17148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:09.456995 17148 hybrid_clock.cc:648] HybridClock initialized: now 1786372329456948 us; error 30 us; skew 500 ppm
I20260810 14:32:09.457300 17148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:09 dist-test-slave-q9h9 krb5kdc[16515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372329, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:09.460248 17148 init.cc:377] Logged in from keytab as oryx/127.6.46.195@KRBTEST.COM (short username oryx)
I20260810 14:32:09.461602 17148 webserver.cc:460] Webserver started at http://127.6.46.195:44247/ using document root <none> and password file <none>
I20260810 14:32:09.462280 17148 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:09.462380 17148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:09.462620 17148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:09.465360 17148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "031348ed4bcd474687b9a337f7cb4378"
format_stamp: "Formatted at 2026-08-10 14:32:09 on dist-test-slave-q9h9"
server_key: "b52bf6fee48da89c88f143a05b9eed73"
server_key_iv: "4a6aa59d911d45c7f84f8410de14d4b4"
server_key_version: "encryptionkey@0"
I20260810 14:32:09.466066 17148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "031348ed4bcd474687b9a337f7cb4378"
format_stamp: "Formatted at 2026-08-10 14:32:09 on dist-test-slave-q9h9"
server_key: "b52bf6fee48da89c88f143a05b9eed73"
server_key_iv: "4a6aa59d911d45c7f84f8410de14d4b4"
server_key_version: "encryptionkey@0"
I20260810 14:32:09.470882 17148 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:32:09.474150 17171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:09.475422 17148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:32:09.475569 17148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "031348ed4bcd474687b9a337f7cb4378"
format_stamp: "Formatted at 2026-08-10 14:32:09 on dist-test-slave-q9h9"
server_key: "b52bf6fee48da89c88f143a05b9eed73"
server_key_iv: "4a6aa59d911d45c7f84f8410de14d4b4"
server_key_version: "encryptionkey@0"
I20260810 14:32:09.475742 17148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:09.498463 17148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:09.501984 17148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:09.502265 17148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:09.503100 17148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:09.504347 17148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:09.504422 17148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:09.504534 17148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:09.504586 17148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:09.535156 17148 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:44219
I20260810 14:32:09.535296 17360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:44219 every 8 connection(s)
I20260810 14:32:09.536353 17148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:32:09.536633 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 17148
I20260810 14:32:09.536844 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:32:09.537402 6331 external_mini_cluster.cc:1428] Setting key 9f01dcd4cea782b6a2db698a71b4c759
Aug 10 14:32:09 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372329, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
I20260810 14:32:09.554502 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:09.538879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:46345 (local address 127.6.46.254:39189)
0810 14:32:09.539115 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:09.539122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:09.540248 (+ 1126us) server_negotiation.cc:407] Connection header received
0810 14:32:09.542068 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:09.542074 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:09.542150 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:09.542226 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:09.543562 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:09.544294 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:09.545666 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:09.546071 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:09.546164 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:09.549226 (+ 3062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:09.549230 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:09.549234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:09.549265 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:09.551769 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:09.552412 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:09.552416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:09.552418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:09.552540 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:09.552920 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:09.552925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:09.552928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:09.553179 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:09.553298 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:09.553910 (+ 612us) server_negotiation.cc:299] Negotiation successful
0810 14:32:09.554163 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 14:32:09.555311 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:09.539328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.195:46345)
0810 14:32:09.540095 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:09.540147 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:09.541819 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:09.542408 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:09.542416 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:09.542883 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:09.543436 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:09.543453 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:09.544397 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:09.544400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:09.545562 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:09.545573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:09.546246 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:09.546251 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:09.546443 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:09.547009 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:09.547028 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:09.549052 (+ 2024us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:09.551897 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:09.551904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:09.551914 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:09.552303 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:09.552647 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:09.552650 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:09.552652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:09.552817 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:09.553315 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:09.553323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:09.553582 (+ 259us) client_negotiation.cc:769] Sending connection context
0810 14:32:09.553803 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 14:32:09.554293 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":305,"threads_started":1}
I20260810 14:32:09.556437 17361 heartbeater.cc:347] Connected to a master server at 127.6.46.254:39189
I20260810 14:32:09.556705 17361 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:09.557336 17361 heartbeater.cc:511] Master 127.6.46.254:39189 requested a full tablet report, sending...
I20260810 14:32:09.558637 16580 ts_manager.cc:194] Registered new tserver with Master: 031348ed4bcd474687b9a337f7cb4378 (127.6.46.195:44219)
I20260810 14:32:09.559405 16580 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:46345
I20260810 14:32:09.564254 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.46.253@KRBTEST.COM" created.
I20260810 14:32:09.599145 16902 heartbeater.cc:503] Master 127.6.46.254:39189 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.253.keytab.
Entry for principal oryx/127.6.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.253.keytab.
I20260810 14:32:09.613530 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.253:35217
--webserver_interface=127.6.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.253.keytab
--principal=oryx/127.6.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:39189,127.6.46.253:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:09.924309 17372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:09.924579 17372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:09.924739 17372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:09.931638 17372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:32:09.931727 17372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:09.931778 17372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:09.931833 17372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:32:09.931875 17372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:32:09.941001 17372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:39189,127.6.46.253:35217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.253.keytab
--principal=oryx/127.6.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.253:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
W20260810 14:32:09.941259 17372 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.46.254:39189,127.6.46.253:35217'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 14:32:09.942164 17372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:09.943331 17372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:09.954041 17377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:09.954000 17382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:09.953956 17379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:09.954669 17372 server_base.cc:1034] running on GCE node
I20260810 14:32:09.955358 17372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:09.956544 17372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:09.957752 17372 hybrid_clock.cc:648] HybridClock initialized: now 1786372329957693 us; error 39 us; skew 500 ppm
I20260810 14:32:09.958069 17372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:09 dist-test-slave-q9h9 krb5kdc[16515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372329, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:09.960903 17372 init.cc:377] Logged in from keytab as oryx/127.6.46.253@KRBTEST.COM (short username oryx)
I20260810 14:32:09.962291 17372 webserver.cc:460] Webserver started at http://127.6.46.253:33675/ using document root <none> and password file <none>
I20260810 14:32:09.962895 17372 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:09.962993 17372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:09.963275 17372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:09.965987 17372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/data/instance:
uuid: "5374e2ca086d4a858725cd7980b705d4"
format_stamp: "Formatted at 2026-08-10 14:32:09 on dist-test-slave-q9h9"
server_key: "9dbe7f67f98f1ac41fe3daa496ccb5d6"
server_key_iv: "5164594a1b0adf638c6e1576823dc09d"
server_key_version: "encryptionkey@0"
I20260810 14:32:09.966791 17372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/wal/instance:
uuid: "5374e2ca086d4a858725cd7980b705d4"
format_stamp: "Formatted at 2026-08-10 14:32:09 on dist-test-slave-q9h9"
server_key: "9dbe7f67f98f1ac41fe3daa496ccb5d6"
server_key_iv: "5164594a1b0adf638c6e1576823dc09d"
server_key_version: "encryptionkey@0"
I20260810 14:32:09.971361 17372 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:32:09.974578 17395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:09.975845 17372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:32:09.975993 17372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/wal
uuid: "5374e2ca086d4a858725cd7980b705d4"
format_stamp: "Formatted at 2026-08-10 14:32:09 on dist-test-slave-q9h9"
server_key: "9dbe7f67f98f1ac41fe3daa496ccb5d6"
server_key_iv: "5164594a1b0adf638c6e1576823dc09d"
server_key_version: "encryptionkey@0"
I20260810 14:32:09.976176 17372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:10.004323 17372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:10.007885 17372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:10.008113 17372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:10.024802 17372 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.253:35217
I20260810 14:32:10.024860 17478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.253:35217 every 8 connection(s)
I20260810 14:32:10.026055 17372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/data/info.pb
I20260810 14:32:10.029461 17480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:10.031124 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 17372
I20260810 14:32:10.038693 17480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:32:10 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372329, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.253@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
Aug 10 14:32:10 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.46.253@KRBTEST.COM
I20260810 14:32:10.056164 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.041207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41910 (local address 127.6.46.254:39189)
0810 14:32:10.041396 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.041402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.042004 (+ 602us) server_negotiation.cc:407] Connection header received
0810 14:32:10.042198 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.042204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.042266 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.042343 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.043342 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.044041 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.045268 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.045677 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.045784 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.049750 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.049757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.049762 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.049800 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.052551 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.053475 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.053481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.053484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.053643 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.054046 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.054050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.054053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.054458 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.054604 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.055330 (+ 726us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.055543 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:32:10.056962 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.041185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:41910)
0810 14:32:10.041860 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.041909 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.042078 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.042413 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.042420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.042971 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:10.043206 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.043216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.044160 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.044164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.045140 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.045151 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.045878 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.045885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.046059 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.046723 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:10.046753 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:10.049542 (+ 2789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:10.052686 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.052696 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.052707 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.053368 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.053738 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.053741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.053743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.053900 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.054577 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:10.054584 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:10.054972 (+ 388us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.055248 (+ 276us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.055697 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":282,"threads_started":1}
I20260810 14:32:10.056981 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.033339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.6.46.253:35217)
0810 14:32:10.034077 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.034088 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.034129 (+ 41us) server_negotiation.cc:407] Connection header received
0810 14:32:10.035681 (+ 1552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.035701 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.036052 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.036352 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.038127 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.039496 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.041139 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.041906 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.042091 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.050289 (+ 8198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.050303 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.050316 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.050352 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.053338 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.054088 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.054094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.054096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.054228 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.054608 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.054612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.054614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.055050 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.055362 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.055592 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.055788 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":310,"threads_started":1}
I20260810 14:32:10.059705 17480 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:32:10 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372329, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.253@KRBTEST.COM for oryx/127.6.46.253@KRBTEST.COM
I20260810 14:32:10.070310 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.060383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:35217 (local address 127.0.0.1:38460)
0810 14:32:10.060548 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.060563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.060633 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.060860 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.060863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.061137 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:10.061261 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.061268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.062132 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.062135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.062974 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.062986 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.063584 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.063588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.063725 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.064127 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:10.064141 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:10.065957 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:10.068455 (+ 2498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.068458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.068460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.068810 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.069117 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.069120 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.069122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.069238 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.069656 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:10.069660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:10.069771 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.069873 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.070094 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 14:32:10.070472 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.060423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38460 (local address 127.6.46.253:35217)
0810 14:32:10.060582 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.060585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.060599 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:32:10.060719 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.060722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.060769 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.060838 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.061365 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.061982 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.063099 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.063472 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.063563 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.066123 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.066128 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.066130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.066157 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.068347 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.068899 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.068902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.068904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.069020 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.069319 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.069322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.069324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.069562 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.069659 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.069948 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.070213 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":52}
I20260810 14:32:10.074285 17138 heartbeater.cc:503] Master 127.6.46.254:39189 was elected leader, sending a full tablet report...
I20260810 14:32:10.074878 17480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: Bootstrap starting.
I20260810 14:32:10.077899 17480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:10.078876 17480 log.cc:824] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:10.081301 17480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: No bootstrap required, opened a new log
I20260810 14:32:10.087905 17480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } }
I20260810 14:32:10.088174 17480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:10.088246 17480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5374e2ca086d4a858725cd7980b705d4, State: Initialized, Role: FOLLOWER
I20260810 14:32:10.088688 17480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } }
I20260810 14:32:10.089413 17496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } } }
I20260810 14:32:10.089721 17496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:32:10.090294 17480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 14:32:10.098512 17516 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:32:10.098664 17516 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:32:10.102057 17372 master_runner.cc:428] Detected that this master 5374e2ca086d4a858725cd7980b705d4 is joining an existing cluster
I20260810 14:32:10.102154 17372 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.46.253:35217
I20260810 14:32:10.122108 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.106109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38474 (local address 127.6.46.253:35217)
0810 14:32:10.106300 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.106306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.107678 (+ 1372us) server_negotiation.cc:407] Connection header received
0810 14:32:10.107830 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.107835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.107888 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.107969 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.109085 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.110232 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.112381 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.113068 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.113165 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.115638 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.115644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.115648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.115683 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.119130 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.119908 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.119913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.119917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.120044 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.120499 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.120505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.120508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.120883 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.121025 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.121508 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.121717 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:32:10.123220 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.106144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.6.46.254:39189)
0810 14:32:10.106478 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.106483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.107137 (+ 654us) server_negotiation.cc:407] Connection header received
0810 14:32:10.107359 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.107365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.107440 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.107508 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.109151 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.110252 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.112829 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.113424 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.113512 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.116213 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.116448 (+ 235us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.116454 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.116498 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.120048 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.120775 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.120780 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.120784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.120930 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.121365 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.121370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.121373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.121750 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.121862 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.122695 (+ 833us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.122905 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260810 14:32:10.123234 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.106833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:35217 (local address 127.0.0.1:38474)
0810 14:32:10.107590 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.107604 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.107717 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.108008 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.108012 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.108563 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:10.108949 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.108964 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.110438 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.110442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.112193 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.112256 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.113463 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.113470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.113626 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.114284 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:10.114307 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:10.115452 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:10.119276 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.119282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.119286 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.119780 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.120194 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.120199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.120202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.120385 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.121101 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:10.121105 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:10.121255 (+ 150us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.122447 (+ 1192us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.122732 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":103,"spinlock_wait_cycles":28544,"thread_start_us":386,"threads_started":1}
I20260810 14:32:10.124464 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.106289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:41914)
0810 14:32:10.107022 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.107042 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.107181 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.107748 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.107752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.108462 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:10.109016 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.109031 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.110415 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.110420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.112674 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.112695 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.113795 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.113802 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.114287 (+ 485us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.114896 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:10.114919 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:10.115981 (+ 1062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:10.120194 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.120199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.120202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.120635 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.121063 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.121068 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.121071 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.121229 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.121959 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:10.121963 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:10.122178 (+ 215us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.123762 (+ 1584us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.124036 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":22,"thread_start_us":410,"threads_started":1}
Aug 10 14:32:10 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372327, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.254@KRBTEST.COM for oryx/127.6.46.253@KRBTEST.COM
WARNING: no policy specified for oryx/127.6.46.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.46.252@KRBTEST.COM" created.
I20260810 14:32:10.146377 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.129846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.6.46.253:35217)
0810 14:32:10.129983 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.129988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.130573 (+ 585us) server_negotiation.cc:407] Connection header received
0810 14:32:10.130909 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.130914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.131020 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.131130 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.132327 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.133389 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.135129 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.135814 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.135944 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.139940 (+ 3996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.139948 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.139953 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.139990 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.143558 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.144424 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.144430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.144434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.144578 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.144958 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.144962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.144964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.145212 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.145346 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.145800 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.145992 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:32:10.146435 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.129843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:35217 (local address 127.0.0.1:38482)
0810 14:32:10.130453 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.130490 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.130704 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.131181 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.131188 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.131915 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:10.132161 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.132174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.133534 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.133539 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.134974 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.134990 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.135991 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.135997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.136197 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.136864 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:10.136889 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:10.139718 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:10.143710 (+ 3992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.143722 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.143730 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.144291 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.144698 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.144703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.144707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.144844 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.145337 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:10.145344 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:10.145502 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.145676 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.145981 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":259,"threads_started":1}
I20260810 14:32:10.148368 16580 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.46.253:35217: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5374e2ca086d4a858725cd7980b705d4"
member_type: NON_VOTER
last_known_addr {
host: "127.6.46.253"
port: 35217
}
attrs {
promote: true
}
}
dest_uuid: "bd97ad2506a34450b6cf70414423528c"
cas_config_opid_index: -1
I20260810 14:32:10.149735 16580 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } }
I20260810 14:32:10.165236 17438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:10.169348 17438 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 FOLLOWER]: Refusing update from remote peer bd97ad2506a34450b6cf70414423528c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 14:32:10.170970 17543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:10.170964 17545 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5374e2ca086d4a858725cd7980b705d4 (127.6.46.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } } }
I20260810 14:32:10.175381 17545 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.46.253:35217
I20260810 14:32:10.177337 17372 master.cc:537] Master@127.6.46.253:35217 shutting down...
I20260810 14:32:10.178893 17543 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } } }
I20260810 14:32:10.179332 17543 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.252.keytab.
Entry for principal oryx/127.6.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.252.keytab.
I20260810 14:32:10.185381 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.252:38073
--webserver_interface=127.6.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.252.keytab
--principal=oryx/127.6.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:39189,127.6.46.253:35217,127.6.46.252:38073
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:32:10.205161 17438 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5374e2ca086d4a858725cd7980b705d4 (127.6.46.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } } }
I20260810 14:32:10.210422 17557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } } }
I20260810 14:32:10.210918 17557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:32:10.211333 17553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } } }
I20260810 14:32:10.211702 17553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:32:10.212078 17496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd97ad2506a34450b6cf70414423528c. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } } }
I20260810 14:32:10.212308 17496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:32:10.236208 17372 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 LEARNER]: Raft consensus shutting down.
I20260810 14:32:10.236634 17372 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 LEARNER]: Raft consensus is shut down!
I20260810 14:32:10.236804 17372 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: stopping tablet replica
I20260810 14:32:10.245011 17372 master.cc:552] Master@127.6.46.253:35217 shutdown complete.
I20260810 14:32:10.245203 17372 master_runner.cc:305] Clearing existing system tablet
I20260810 14:32:10.245769 17372 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:10.245944 17372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:10.249603 17372 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:32:10.254377 17563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:10.254745 17372 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:10.255234 17372 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260810 14:32:10.255396 17372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/wal
uuid: "5374e2ca086d4a858725cd7980b705d4"
format_stamp: "Formatted at 2026-08-10 14:32:09 on dist-test-slave-q9h9"
server_key: "9dbe7f67f98f1ac41fe3daa496ccb5d6"
server_key_iv: "5164594a1b0adf638c6e1576823dc09d"
server_key_version: "encryptionkey@0"
I20260810 14:32:10.260474 17372 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:10.267482 17372 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 14:32:10.267745 17372 log.cc:1196] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 14:32:10.268323 17372 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: Deleting consensus metadata
I20260810 14:32:10.268638 17372 master_runner.cc:315] Copying system tablet from 127.6.46.254:39189
I20260810 14:32:10.275442 17372 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: tablet copy: Beginning tablet copy session from remote peer at address 127.6.46.254:39189
I20260810 14:32:10.291673 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.276629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:41920)
0810 14:32:10.277303 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.277328 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.277492 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.277897 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.277902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.278472 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:10.278752 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.278766 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.280251 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.280257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.281778 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.281795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.282838 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.282846 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.283036 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.283723 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:10.283745 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:10.284815 (+ 1070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:10.288625 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.288630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.288634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.289116 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.289520 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.289525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.289529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.289702 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.290343 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:10.290349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:10.290534 (+ 185us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.290731 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.291122 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":376,"threads_started":1}
I20260810 14:32:10.292660 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.276662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41920 (local address 127.6.46.254:39189)
0810 14:32:10.276860 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.276867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.277439 (+ 572us) server_negotiation.cc:407] Connection header received
0810 14:32:10.277630 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.277636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.277738 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.277850 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.278919 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.280100 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.281943 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.282636 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.282761 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.285035 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.285043 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.285048 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.285089 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.288483 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.289257 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.289263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.289266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.289408 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.289817 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.289822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.289825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.290210 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.290377 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.292066 (+ 1689us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.292337 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 14:32:10.293598 16623 tablet_copy_service.cc:140] P bd97ad2506a34450b6cf70414423528c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5374e2ca086d4a858725cd7980b705d4 ({username='oryx', principal='oryx/127.6.46.253@KRBTEST.COM'} at 127.0.0.1:41920)
I20260810 14:32:10.293999 16623 tablet_copy_service.cc:161] P bd97ad2506a34450b6cf70414423528c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5374e2ca086d4a858725cd7980b705d4 at {username='oryx', principal='oryx/127.6.46.253@KRBTEST.COM'} at 127.0.0.1:41920: session id = 5374e2ca086d4a858725cd7980b705d4-00000000000000000000000000000000
I20260810 14:32:10.298960 16623 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 14:32:10.301602 17372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 14:32:10.308564 17372 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: tablet copy: Starting download of 0 data blocks...
I20260810 14:32:10.308956 17372 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: tablet copy: Starting download of 1 WAL segments...
I20260810 14:32:10.312628 17372 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 14:32:10.317184 16623 tablet_copy_service.cc:342] P bd97ad2506a34450b6cf70414423528c: Request end of tablet copy session 5374e2ca086d4a858725cd7980b705d4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.46.253@KRBTEST.COM'} at 127.0.0.1:41920
I20260810 14:32:10.317503 16623 tablet_copy_service.cc:433] P bd97ad2506a34450b6cf70414423528c: ending tablet copy session 5374e2ca086d4a858725cd7980b705d4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5374e2ca086d4a858725cd7980b705d4
I20260810 14:32:10.329480 17372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:10.332026 17577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:10.335184 17578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:10.342239 17372 server_base.cc:1034] running on GCE node
W20260810 14:32:10.343134 17581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:10.343617 17372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:10.344244 17372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:10.345508 17372 hybrid_clock.cc:648] HybridClock initialized: now 1786372330345442 us; error 30 us; skew 500 ppm
I20260810 14:32:10.345736 17372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:10 dist-test-slave-q9h9 krb5kdc[16515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372330, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:10.348017 17372 init.cc:377] Logged in from keytab as oryx/127.6.46.253@KRBTEST.COM (short username oryx)
I20260810 14:32:10.350369 17372 webserver.cc:460] Webserver started at http://127.6.46.253:40753/ using document root <none> and password file <none>
I20260810 14:32:10.350737 17372 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:10.350835 17372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:10.354144 17372 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 14:32:10.356178 17589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:10.356614 17372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260810 14:32:10.356796 17372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/wal
uuid: "5374e2ca086d4a858725cd7980b705d4"
format_stamp: "Formatted at 2026-08-10 14:32:09 on dist-test-slave-q9h9"
server_key: "9dbe7f67f98f1ac41fe3daa496ccb5d6"
server_key_iv: "5164594a1b0adf638c6e1576823dc09d"
server_key_version: "encryptionkey@0"
I20260810 14:32:10.357039 17372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:10.425890 17372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:10.430734 17372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:10.456343 17372 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.253:35217
I20260810 14:32:10.456400 17671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.253:35217 every 8 connection(s)
I20260810 14:32:10.457510 17372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-1/data/info.pb
I20260810 14:32:10.463510 17673 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:32:10.467756 17673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: Bootstrap starting.
I20260810 14:32:10.484104 17673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:32:10.484777 17673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: Bootstrap complete.
I20260810 14:32:10.485957 17673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } }
I20260810 14:32:10.486254 17673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5374e2ca086d4a858725cd7980b705d4, State: Initialized, Role: LEARNER
I20260810 14:32:10.486536 17673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } }
I20260810 14:32:10.487422 17683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } } }
I20260810 14:32:10.487721 17673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:32:10.487897 17683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:32:10.500654 17699 catalog_manager.cc:1259] Loaded cluster ID: dfcf555871a444528a1ebb2de1eda106
I20260810 14:32:10.500756 17699 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: loading cluster ID for follower catalog manager: success
I20260810 14:32:10.504521 17699 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: acquiring CA information for follower catalog manager: success
I20260810 14:32:10.507623 17699 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:32:10.561604 17361 heartbeater.cc:503] Master 127.6.46.254:39189 was elected leader, sending a full tablet report...
W20260810 14:32:10.608800 17552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:10.609057 17552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:10.609143 17552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:10.616812 17552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:32:10.616902 17552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:10.616947 17552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:10.617002 17552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:32:10.617045 17552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:32:10.628013 17552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:39189,127.6.46.253:35217,127.6.46.252:38073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.252.keytab
--principal=oryx/127.6.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.252:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:10.629344 17552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:10.630671 17552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:10.643538 17709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:10.643668 17711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:10.646322 17718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:10.647678 17552 server_base.cc:1034] running on GCE node
I20260810 14:32:10.648372 17552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:10.649751 17552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:10.650950 17552 hybrid_clock.cc:648] HybridClock initialized: now 1786372330650898 us; error 42 us; skew 500 ppm
I20260810 14:32:10.651273 17552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:10 dist-test-slave-q9h9 krb5kdc[16515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372330, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:10.655181 17552 init.cc:377] Logged in from keytab as oryx/127.6.46.252@KRBTEST.COM (short username oryx)
I20260810 14:32:10.656852 17552 webserver.cc:460] Webserver started at http://127.6.46.252:34921/ using document root <none> and password file <none>
I20260810 14:32:10.657755 17552 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:10.657961 17552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:10.658396 17552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:10.661314 17552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/data/instance:
uuid: "f3092a8b44094e42b160704fc7b67d66"
format_stamp: "Formatted at 2026-08-10 14:32:10 on dist-test-slave-q9h9"
server_key: "9c5aadda1be190537d7f1e6f6e50036c"
server_key_iv: "70989fa4cada7328d1afd4663c3b1e14"
server_key_version: "encryptionkey@0"
I20260810 14:32:10.662267 17552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/wal/instance:
uuid: "f3092a8b44094e42b160704fc7b67d66"
format_stamp: "Formatted at 2026-08-10 14:32:10 on dist-test-slave-q9h9"
server_key: "9c5aadda1be190537d7f1e6f6e50036c"
server_key_iv: "70989fa4cada7328d1afd4663c3b1e14"
server_key_version: "encryptionkey@0"
I20260810 14:32:10.668030 17552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 14:32:10.671842 17729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:10.673007 17552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:32:10.673174 17552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/wal
uuid: "f3092a8b44094e42b160704fc7b67d66"
format_stamp: "Formatted at 2026-08-10 14:32:10 on dist-test-slave-q9h9"
server_key: "9c5aadda1be190537d7f1e6f6e50036c"
server_key_iv: "70989fa4cada7328d1afd4663c3b1e14"
server_key_version: "encryptionkey@0"
I20260810 14:32:10.673406 17552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:10.709549 17552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:10.711556 17543 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } } }
I20260810 14:32:10.711887 17543 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:10.715525 17552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:10.715855 17552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:10.721057 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.712647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.6.46.253:35217)
0810 14:32:10.713881 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.713886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.713903 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:32:10.713965 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.713969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.714080 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.714185 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:10.715018 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.716295 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.719182 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.720149 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.720248 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.720392 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.720725 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.720826 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":980,"threads_started":1}
I20260810 14:32:10.721174 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.712201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:35217 (local address 127.0.0.1:38496)
0810 14:32:10.712649 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.712670 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.712806 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.714258 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.714264 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.714357 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:10.714836 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.714853 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.716430 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.716435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.718977 (+ 2542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.718995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.720325 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.720331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.720521 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.720601 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.720713 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.720855 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":663,"threads_started":1}
I20260810 14:32:10.722496 17683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd97ad2506a34450b6cf70414423528c. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } } }
I20260810 14:32:10.722855 17683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:32:10.723363 17543 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c: attempting to promote NON_VOTER 5374e2ca086d4a858725cd7980b705d4 to VOTER
I20260810 14:32:10.724390 17543 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } }
I20260810 14:32:10.726516 17631 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 LEARNER]: Refusing update from remote peer bd97ad2506a34450b6cf70414423528c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 14:32:10.727293 17737 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 14:32:10.731051 17543 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5374e2ca086d4a858725cd7980b705d4 (127.6.46.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } }
I20260810 14:32:10.731873 17631 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5374e2ca086d4a858725cd7980b705d4 (127.6.46.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } }
I20260810 14:32:10.735109 17543 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } }
I20260810 14:32:10.735520 17543 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:10.735359 17736 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } }
I20260810 14:32:10.736065 17736 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:10.736295 17683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } }
I20260810 14:32:10.736469 17683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:32:10.747902 17552 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.252:38073
I20260810 14:32:10.747942 17827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.252:38073 every 8 connection(s)
I20260810 14:32:10.749270 17552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/data/info.pb
I20260810 14:32:10.753263 17829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:10.753816 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 17552
I20260810 14:32:10.762367 17829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:32:10 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.46.252@KRBTEST.COM
Aug 10 14:32:10 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372330, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.252@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
I20260810 14:32:10.777952 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.755907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58830 (local address 127.6.46.252:38073)
0810 14:32:10.757208 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.757219 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.757260 (+ 41us) server_negotiation.cc:407] Connection header received
0810 14:32:10.759834 (+ 2574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.759856 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.760229 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.760582 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.762072 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.763415 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.765015 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.765966 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.766174 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.770854 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.770865 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.770877 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.770922 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.774294 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.774959 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.774965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.774971 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.775118 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.775460 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.775463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.775466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.775909 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.776198 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.776594 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.776957 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1086,"thread_start_us":793,"threads_started":1}
I20260810 14:32:10.779793 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.764724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.6.46.254:39189)
0810 14:32:10.764888 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.764894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.765971 (+ 1077us) server_negotiation.cc:407] Connection header received
0810 14:32:10.766241 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.766246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.766309 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.766390 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.767855 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.768562 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.770306 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.770840 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.770971 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.774554 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.774559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.774562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.774592 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.777178 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.777856 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.777860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.777862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.777975 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.778418 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.778421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.778423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.778697 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.778848 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.779376 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.779551 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:32:10.779969 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.764668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:41926)
0810 14:32:10.765827 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.765862 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.766114 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.766526 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.766534 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.767348 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:10.767712 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.767722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.768745 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.768748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.770167 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.770184 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.771045 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.771049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.771199 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.771782 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:10.771806 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:10.774398 (+ 2592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:10.777305 (+ 2907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.777316 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.777326 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.777737 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.778095 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.778100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.778103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.778270 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.778840 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:10.778847 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:10.779020 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.779259 (+ 239us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.779574 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"thread_start_us":500,"threads_started":1}
I20260810 14:32:10.781292 17829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:32:10 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372330, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.252@KRBTEST.COM for oryx/127.6.46.253@KRBTEST.COM
I20260810 14:32:10.792990 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.781943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:35217 (local address 127.0.0.1:38510)
0810 14:32:10.782127 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.782141 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.782225 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.782477 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.782479 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.782751 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:10.782870 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.782877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.783782 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.783785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.784656 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.784666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.785238 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.785242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.785370 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.785834 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:10.785850 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:10.788018 (+ 2168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:10.791247 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.791251 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.791254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.791596 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.791887 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.791890 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.791892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.792007 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.792400 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:10.792403 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:10.792504 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.792600 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.792794 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 14:32:10.793131 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.781943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.6.46.253:35217)
0810 14:32:10.782161 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.782165 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.782185 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:32:10.782315 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.782319 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.782388 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.782459 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.782971 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.783685 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.784756 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.785135 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.785222 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.788178 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.788184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.788187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.788220 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.791132 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.791684 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.791688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.791690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.791810 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.792077 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.792079 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.792081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.792323 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.792426 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.792704 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.792893 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:32:10.793963 17829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:32:10 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372330, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.252@KRBTEST.COM for oryx/127.6.46.252@KRBTEST.COM
I20260810 14:32:10.806612 17833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.794554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:38073 (local address 127.0.0.1:58842)
0810 14:32:10.794781 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.794800 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.794879 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.795174 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.795178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.795602 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:10.795733 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.795741 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.796599 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.796604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.797386 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.797396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.798110 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.798116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.798392 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.798948 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:10.798963 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:10.801196 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:10.804233 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.804238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.804242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.804661 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.804979 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.804984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.804988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.805135 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.805679 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:10.805684 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:10.805812 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.805922 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.806175 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 14:32:10.806819 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.794813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58842 (local address 127.6.46.252:38073)
0810 14:32:10.794974 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.794978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.794990 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:32:10.795020 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.795023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.795066 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.795138 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.795871 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.796484 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.797510 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.797908 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.798040 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.801347 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.801351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.801354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.801379 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.804120 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.804762 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.804767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.804769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.804867 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.805231 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.805236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.805239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.805562 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.805696 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.806080 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.806338 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":44}
I20260810 14:32:10.811805 17829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: Bootstrap starting.
I20260810 14:32:10.814865 17829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:10.815809 17829 log.cc:824] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:10.818440 17829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: No bootstrap required, opened a new log
I20260810 14:32:10.825359 17829 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } }
I20260810 14:32:10.825791 17829 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:10.825973 17829 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3092a8b44094e42b160704fc7b67d66, State: Initialized, Role: FOLLOWER
I20260810 14:32:10.826512 17829 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } }
I20260810 14:32:10.827451 17837 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } } }
I20260810 14:32:10.828079 17837 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:32:10.828228 17829 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 14:32:10.837114 17850 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:32:10.837231 17850 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:32:10.841341 17552 master_runner.cc:428] Detected that this master f3092a8b44094e42b160704fc7b67d66 is joining an existing cluster
I20260810 14:32:10.841470 17552 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.46.252:38073
I20260810 14:32:10.861527 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.845307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.6.46.254:39189)
0810 14:32:10.845487 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.845494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.846699 (+ 1205us) server_negotiation.cc:407] Connection header received
0810 14:32:10.846862 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.846869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.846954 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.847057 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.848317 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.849515 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.851520 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.852236 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.852370 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.854863 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.854870 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.854875 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.854914 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.858401 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.859207 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.859213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.859217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.859374 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.859870 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.859876 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.859879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.860214 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.860345 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.860965 (+ 620us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.861186 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:32:10.862601 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.845723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:41930)
0810 14:32:10.846578 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.846597 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.846744 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.847098 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.847106 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.847810 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:10.848170 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.848184 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.849687 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.849693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.851370 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.851385 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.852453 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.852460 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.852647 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.853341 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:10.853363 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:10.854631 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:10.858572 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.858576 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.858580 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.859058 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.859526 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.859531 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.859535 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.859734 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.860431 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:10.860436 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:10.860664 (+ 228us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.861762 (+ 1098us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.862104 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":556,"threads_started":1}
I20260810 14:32:10.869359 17743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.845372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.6.46.253:35217)
0810 14:32:10.845566 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.845572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.852689 (+ 7117us) server_negotiation.cc:407] Connection header received
0810 14:32:10.852831 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.852836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.852910 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.852999 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.854558 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.855732 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.858062 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.858735 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.858831 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.861375 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.861382 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.861387 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.861438 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.865514 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.866470 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.866476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.866479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.866724 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.867465 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.867470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.867474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.867938 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.868107 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.868795 (+ 688us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.869030 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 14:32:10.870426 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.845723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:35217 (local address 127.0.0.1:38512)
0810 14:32:10.852591 (+ 6868us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.852605 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.852721 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.853165 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.853170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.853960 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:10.854373 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.854386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.856120 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.856126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.857875 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.857890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.859125 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.859131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.859307 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.859949 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:10.859973 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:10.861165 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:10.865681 (+ 4516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.865686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.865690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.866321 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.866989 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.866994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.866998 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.867327 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.868195 (+ 868us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:10.868199 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:10.868460 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.869636 (+ 1176us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.869939 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6761,"mutex_wait_us":23,"thread_start_us":7045,"threads_started":1}
I20260810 14:32:10.879128 17861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.853528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:38073 (local address 127.0.0.1:58850)
0810 14:32:10.862683 (+ 9155us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.862700 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.862959 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.863319 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.863324 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.864102 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:10.864498 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.864515 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.866396 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.866401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.868183 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.868197 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.869627 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.869638 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.869824 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.870475 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:10.870501 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:10.872738 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:10.876702 (+ 3964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.876706 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.876708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.877060 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.877450 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.877453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.877456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.877597 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.878207 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:10.878216 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:10.878374 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.878571 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.878804 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9028,"thread_start_us":9255,"threads_started":1}
I20260810 14:32:10.879354 17832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.853221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.6.46.252:38073)
0810 14:32:10.853417 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.853422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.862773 (+ 9351us) server_negotiation.cc:407] Connection header received
0810 14:32:10.863086 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.863091 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.863158 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.863261 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.864660 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.866169 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.868323 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.868938 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.869026 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.872924 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.872935 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.872939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.872973 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.876567 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.877167 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.877173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.877176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.877353 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.877698 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.877703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.877706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.878079 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.878248 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.878709 (+ 461us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.878954 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":70}
I20260810 14:32:10.883497 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 16693
Aug 10 14:32:10 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.46.193@KRBTEST.COM
Aug 10 14:32:10 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372327, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.254@KRBTEST.COM for oryx/127.6.46.252@KRBTEST.COM
I20260810 14:32:10.903852 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.884487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.6.46.193:36107)
0810 14:32:10.885180 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.885190 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.885222 (+ 32us) server_negotiation.cc:407] Connection header received
0810 14:32:10.885289 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.885297 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.885557 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.885745 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.887014 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.888583 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.889855 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.890748 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.890878 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.895771 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.895786 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.895803 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.895850 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.899438 (+ 3588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.900638 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.900652 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.900655 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.900778 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.901288 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.901292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.901294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.901742 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.902130 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.903165 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.903433 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":346,"threads_started":1}
I20260810 14:32:10.905550 16896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:10.908500 17863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.869779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.6.46.252:38073)
0810 14:32:10.874471 (+ 4692us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:10.874475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:10.874494 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:32:10.874565 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:10.874568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:10.874622 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:10.874742 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:10.894427 (+ 19685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.895785 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.897033 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.897762 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.897885 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.901596 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:10.901606 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:10.901610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:10.901646 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:10.905412 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.906314 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.906320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.906323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.906467 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:10.906881 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:10.906886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:10.906889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:10.907232 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:10.907344 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:10.907925 (+ 581us) server_negotiation.cc:299] Negotiation successful
0810 14:32:10.908120 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4595,"thread_start_us":4453,"threads_started":1}
I20260810 14:32:10.909365 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:10.869132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:38073 (local address 127.0.0.1:58862)
0810 14:32:10.869466 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:10.869484 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:10.869579 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:10.874779 (+ 5200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:10.874785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:10.875190 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:10.875319 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.875327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.895913 (+ 20586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.895919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.896900 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:10.896915 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:10.897911 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:10.897918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:10.898130 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:10.898686 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:10.898709 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:10.901269 (+ 2560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:10.905530 (+ 4261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.905534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.905537 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.906182 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.906582 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:10.906588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:10.906591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:10.906754 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:10.907437 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:10.907442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:10.907656 (+ 214us) client_negotiation.cc:769] Sending connection context
0810 14:32:10.908736 (+ 1080us) client_negotiation.cc:240] Negotiation successful
0810 14:32:10.909028 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260810 14:32:10.912696 16580 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.46.252:38073: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f3092a8b44094e42b160704fc7b67d66"
member_type: NON_VOTER
last_known_addr {
host: "127.6.46.252"
port: 38073
}
attrs {
promote: true
}
}
dest_uuid: "bd97ad2506a34450b6cf70414423528c"
cas_config_opid_index: 6
I20260810 14:32:10.913491 16580 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: NON_VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: true } }
I20260810 14:32:10.915970 17631 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 FOLLOWER]: Refusing update from remote peer bd97ad2506a34450b6cf70414423528c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 14:32:10.916687 17736 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 14:32:10.916815 17785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:10.920054 17543 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f3092a8b44094e42b160704fc7b67d66 (127.6.46.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: NON_VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: true } } }
I20260810 14:32:10.920569 17785 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 FOLLOWER]: Refusing update from remote peer bd97ad2506a34450b6cf70414423528c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 14:32:10.920603 17631 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f3092a8b44094e42b160704fc7b67d66 (127.6.46.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: NON_VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: true } } }
I20260810 14:32:10.921316 17737 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: NON_VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:10.924372 17683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: NON_VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: true } } }
I20260810 14:32:10.924639 17683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:32:10.924773 17543 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.46.252:38073
I20260810 14:32:10.925467 17552 master.cc:537] Master@127.6.46.252:38073 shutting down...
I20260810 14:32:10.925678 17543 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: NON_VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: true } } }
I20260810 14:32:10.925890 17543 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:10.926210 17737 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: NON_VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: true } } }
I20260810 14:32:10.926437 17737 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:10.937503 17785 raft_consensus.cc:858] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } }
W20260810 14:32:10.937768 17785 raft_consensus.cc:1565] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } }
I20260810 14:32:10.949899 17785 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5374e2ca086d4a858725cd7980b705d4 (127.6.46.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } } }
I20260810 14:32:10.954188 17878 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } } }
I20260810 14:32:10.954564 17878 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 14:32:10.954670 17871 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } } }
I20260810 14:32:10.955076 17871 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 14:32:10.954978 17837 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd97ad2506a34450b6cf70414423528c. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: NON_VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: true } } }
I20260810 14:32:10.955175 17837 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260810 14:32:10.956948 17543 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: NON_VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: true } } }
I20260810 14:32:10.957223 17543 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:10.957695 17552 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
W20260810 14:32:10.957744 16562 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260810 14:32:10.958108 17552 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260810 14:32:10.958277 17552 tablet_replica.cc:330] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: stopping tablet replica
I20260810 14:32:10.966647 17552 master.cc:552] Master@127.6.46.252:38073 shutdown complete.
I20260810 14:32:10.966763 17552 master_runner.cc:305] Clearing existing system tablet
I20260810 14:32:10.967165 17552 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:10.967326 17552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260810 14:32:10.987079 16562 consensus_peers.cc:487] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c -> Peer f3092a8b44094e42b160704fc7b67d66 (127.6.46.252:38073): Couldn't send request to peer f3092a8b44094e42b160704fc7b67d66. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260810 14:32:10.989795 17552 fs_manager.cc:709] Time spent opening directory manager: real 0.022s user 0.009s sys 0.013s
I20260810 14:32:11.002429 17883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:11.002871 17552 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:11.003108 17552 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.005s sys 0.006s
I20260810 14:32:11.003279 17552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/wal
uuid: "f3092a8b44094e42b160704fc7b67d66"
format_stamp: "Formatted at 2026-08-10 14:32:10 on dist-test-slave-q9h9"
server_key: "9c5aadda1be190537d7f1e6f6e50036c"
server_key_iv: "70989fa4cada7328d1afd4663c3b1e14"
server_key_version: "encryptionkey@0"
I20260810 14:32:11.008056 17552 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:11.014566 17552 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 14:32:11.014712 17552 log.cc:1196] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 14:32:11.015172 17552 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: Deleting consensus metadata
I20260810 14:32:11.015420 17552 master_runner.cc:315] Copying system tablet from 127.6.46.254:39189
I20260810 14:32:11.018198 17552 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: tablet copy: Beginning tablet copy session from remote peer at address 127.6.46.254:39189
I20260810 14:32:11.040565 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 16693
I20260810 14:32:11.043695 16905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:11.018975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.6.46.254:39189)
0810 14:32:11.019210 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:11.019217 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:11.026615 (+ 7398us) server_negotiation.cc:407] Connection header received
0810 14:32:11.026792 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:11.026798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:11.026894 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:11.027001 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:11.028056 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.029242 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.031234 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.031924 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.032021 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:11.034542 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:11.034549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:11.034554 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:11.034593 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:11.038148 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:11.039175 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:11.039180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:11.039183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:11.039350 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:11.039850 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:11.039855 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:11.039859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:11.040205 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:11.040346 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:11.040897 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 14:32:11.043344 (+ 2447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 14:32:11.044652 17893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:11.018934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:41944)
0810 14:32:11.026481 (+ 7547us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:11.026501 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:11.026655 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:11.027049 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:11.027054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:11.027617 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:11.027891 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:11.027904 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.029408 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.029414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:11.031061 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:11.031078 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.032269 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.032275 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:11.032459 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:11.033152 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:11.033174 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:11.034335 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:11.038295 (+ 3960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:11.038300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:11.038305 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:11.039043 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:11.039509 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:11.039514 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:11.039518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:11.039715 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:11.040430 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:11.040435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:11.040619 (+ 184us) client_negotiation.cc:769] Sending connection context
0810 14:32:11.043876 (+ 3257us) client_negotiation.cc:240] Negotiation successful
0810 14:32:11.044214 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7400,"thread_start_us":7273,"threads_started":1}
I20260810 14:32:11.045223 16623 tablet_copy_service.cc:140] P bd97ad2506a34450b6cf70414423528c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f3092a8b44094e42b160704fc7b67d66 ({username='oryx', principal='oryx/127.6.46.252@KRBTEST.COM'} at 127.0.0.1:41944)
I20260810 14:32:11.045393 16623 tablet_copy_service.cc:161] P bd97ad2506a34450b6cf70414423528c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f3092a8b44094e42b160704fc7b67d66 at {username='oryx', principal='oryx/127.6.46.252@KRBTEST.COM'} at 127.0.0.1:41944: session id = f3092a8b44094e42b160704fc7b67d66-00000000000000000000000000000000
I20260810 14:32:11.049010 16623 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 14:32:11.053561 17552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 14:32:11.060097 17552 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: tablet copy: Starting download of 0 data blocks...
I20260810 14:32:11.060403 17552 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: tablet copy: Starting download of 1 WAL segments...
I20260810 14:32:11.062793 17552 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 14:32:11.066831 16623 tablet_copy_service.cc:342] P bd97ad2506a34450b6cf70414423528c: Request end of tablet copy session f3092a8b44094e42b160704fc7b67d66-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.46.252@KRBTEST.COM'} at 127.0.0.1:41944
I20260810 14:32:11.066968 16623 tablet_copy_service.cc:433] P bd97ad2506a34450b6cf70414423528c: ending tablet copy session f3092a8b44094e42b160704fc7b67d66-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f3092a8b44094e42b160704fc7b67d66
I20260810 14:32:11.070065 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:36107
--local_ip_for_outbound_sockets=127.6.46.193
--tserver_master_addrs=127.6.46.254:39189
--webserver_port=33861
--webserver_interface=127.6.46.193
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab
--principal=oryx/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.46.254:39189,127.6.46.253:35217
--tserver_master_addrs=127.6.46.254:39189,127.6.46.253:35217,127.6.46.252:38073 with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:32:11.078421 17552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:11.081075 17896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:11.086551 17899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:11.092816 17905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:11.093494 17552 server_base.cc:1034] running on GCE node
I20260810 14:32:11.093830 17552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:11.094604 17552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:11.095804 17552 hybrid_clock.cc:648] HybridClock initialized: now 1786372331095762 us; error 31 us; skew 500 ppm
I20260810 14:32:11.096072 17552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:11 dist-test-slave-q9h9 krb5kdc[16515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372331, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:11.098212 17552 init.cc:377] Logged in from keytab as oryx/127.6.46.252@KRBTEST.COM (short username oryx)
I20260810 14:32:11.101981 17552 webserver.cc:460] Webserver started at http://127.6.46.252:35225/ using document root <none> and password file <none>
I20260810 14:32:11.102334 17552 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:11.102447 17552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:11.105604 17552 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:32:11.107446 17918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:11.107818 17552 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 14:32:11.107936 17552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/wal
uuid: "f3092a8b44094e42b160704fc7b67d66"
format_stamp: "Formatted at 2026-08-10 14:32:10 on dist-test-slave-q9h9"
server_key: "9c5aadda1be190537d7f1e6f6e50036c"
server_key_iv: "70989fa4cada7328d1afd4663c3b1e14"
server_key_version: "encryptionkey@0"
I20260810 14:32:11.108117 17552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:11.137676 17552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:11.142366 17552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:11.165093 17552 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.252:38073
I20260810 14:32:11.165153 18005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.252:38073 every 8 connection(s)
I20260810 14:32:11.166141 17552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/master-2/data/info.pb
I20260810 14:32:11.171351 18006 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:32:11.175294 18006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: Bootstrap starting.
I20260810 14:32:11.191263 18006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:32:11.191799 18006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: Bootstrap complete.
I20260810 14:32:11.192876 18006 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: NON_VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: true } }
I20260810 14:32:11.193168 18006 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f3092a8b44094e42b160704fc7b67d66, State: Initialized, Role: LEARNER
I20260810 14:32:11.193425 18006 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: NON_VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: true } }
I20260810 14:32:11.194366 18010 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: NON_VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: true } } }
I20260810 14:32:11.194707 18006 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:32:11.194968 18010 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:32:11.208201 18030 catalog_manager.cc:1259] Loaded cluster ID: dfcf555871a444528a1ebb2de1eda106
I20260810 14:32:11.208303 18030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: loading cluster ID for follower catalog manager: success
I20260810 14:32:11.210805 18030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: acquiring CA information for follower catalog manager: success
I20260810 14:32:11.212649 18030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 14:32:11.432337 17894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:11.432623 17894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:11.432760 17894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:11.440243 17894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:11.440359 17894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:11.440498 17894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:32:11.450337 17894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab
--principal=oryx/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:36107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=33861
--tserver_master_addrs=127.6.46.254:39189,127.6.46.253:35217,127.6.46.252:38073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:11.451577 17894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:11.452649 17736 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: NON_VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: true } } }
I20260810 14:32:11.452952 17736 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:11.453241 17894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:32:11.459401 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:11.453316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.6.46.252:38073)
0810 14:32:11.453819 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:11.453824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:11.453839 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:32:11.453897 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:11.453900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:11.453968 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:11.454099 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:11.454531 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.455604 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.457245 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.458210 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.458334 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:11.458573 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:11.458925 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 14:32:11.459068 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":241,"threads_started":1}
I20260810 14:32:11.459725 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:11.452924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:38073 (local address 127.0.0.1:58872)
0810 14:32:11.453465 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:11.453485 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:11.453633 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:11.454118 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:11.454123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:11.454152 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:11.454361 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:11.454372 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.455835 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.455841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:11.457083 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:11.457101 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.458374 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.458380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:11.458754 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:11.458781 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 14:32:11.458924 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 14:32:11.459106 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":269,"threads_started":1}
I20260810 14:32:11.460625 17965 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.7] Dedup: 1.7->[]
I20260810 14:32:11.461225 18010 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd97ad2506a34450b6cf70414423528c. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: NON_VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: true } } }
I20260810 14:32:11.461431 18010 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: This master's current role is: LEARNER
I20260810 14:32:11.461930 17736 raft_consensus.cc:1050] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c: attempting to promote NON_VOTER f3092a8b44094e42b160704fc7b67d66 to VOTER
I20260810 14:32:11.462566 17736 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } }
I20260810 14:32:11.464556 17631 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 FOLLOWER]: Refusing update from remote peer bd97ad2506a34450b6cf70414423528c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 14:32:11.464545 17965 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 LEARNER]: Refusing update from remote peer bd97ad2506a34450b6cf70414423528c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 14:32:11.465103 17543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 14:32:11.465780 17543 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 14:32:11.467635 18035 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f3092a8b44094e42b160704fc7b67d66 (127.6.46.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } } }
I20260810 14:32:11.468482 17631 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f3092a8b44094e42b160704fc7b67d66 (127.6.46.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } } }
I20260810 14:32:11.471364 18035 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } } }
I20260810 14:32:11.471670 18035 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:11.471637 17543 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } } }
I20260810 14:32:11.471921 17543 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:11.474705 18043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } } }
I20260810 14:32:11.475117 18043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:32:11.477110 18035 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } } }
I20260810 14:32:11.477341 18035 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:11.478192 17965 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f3092a8b44094e42b160704fc7b67d66 (127.6.46.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } } }
I20260810 14:32:11.481128 18010 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd97ad2506a34450b6cf70414423528c" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } } }
I20260810 14:32:11.481384 18010 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 14:32:11.482795 18045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:11.483083 18041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:11.483741 18049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:11.484140 17894 server_base.cc:1034] running on GCE node
I20260810 14:32:11.484706 17894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:11.485579 17894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:11.486778 17894 hybrid_clock.cc:648] HybridClock initialized: now 1786372331486732 us; error 36 us; skew 500 ppm
I20260810 14:32:11.487061 17894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:11 dist-test-slave-q9h9 krb5kdc[16515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372331, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:11.490300 17894 init.cc:377] Logged in from keytab as oryx/127.6.46.193@KRBTEST.COM (short username oryx)
I20260810 14:32:11.491595 17894 webserver.cc:460] Webserver started at http://127.6.46.193:33861/ using document root <none> and password file <none>
I20260810 14:32:11.492259 17894 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:11.492381 17894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:11.498426 17894 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:32:11.501217 18059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:11.502527 17894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:32:11.502691 17894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "a6e4c148a686452caad3c9f8cb9bf56c"
format_stamp: "Formatted at 2026-08-10 14:32:08 on dist-test-slave-q9h9"
server_key: "7d5e9f329f3bb6c8157cf7d35ad6dbe5"
server_key_iv: "fe52127590f497ef02f0d2ea22b7822b"
server_key_version: "encryptionkey@0"
I20260810 14:32:11.503616 17894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:11.563994 17894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:11.569551 17894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:11.570003 17894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:11.571451 17894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:11.573266 17894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:11.573359 17894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:11.573452 17894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:11.573534 17894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:11.606679 17894 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:36107
I20260810 14:32:11.606827 18253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:36107 every 8 connection(s)
I20260810 14:32:11.608165 17894 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:32:11.613950 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 17894
I20260810 14:32:11.614320 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 16910
Aug 10 14:32:11 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372331, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
Aug 10 14:32:11 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372331, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.252@KRBTEST.COM
I20260810 14:32:11.644091 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:11.612077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:42747 (local address 127.6.46.252:38073)
0810 14:32:11.612235 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:11.612239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:11.618618 (+ 6379us) server_negotiation.cc:407] Connection header received
0810 14:32:11.618878 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:11.618884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:11.618972 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:11.619050 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:11.621713 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.623922 (+ 2209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.630768 (+ 6846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.631457 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.631563 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:11.636611 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:11.636620 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:11.636625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:11.636693 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:11.640190 (+ 3497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:11.641206 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:11.641211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:11.641214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:11.641365 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:11.641781 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:11.641787 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:11.641790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:11.642158 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:11.642293 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:11.643536 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0810 14:32:11.643764 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:32:11.644961 18263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:11.611222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.193:37511)
0810 14:32:11.612115 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:11.612163 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:11.614878 (+ 2715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:11.622110 (+ 7232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:11.622114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:11.622587 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:11.622887 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:11.622898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.624623 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.624628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:11.626620 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:11.626637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.627651 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.627659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:11.627934 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:11.628719 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:11.628752 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:11.632597 (+ 3845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:11.638346 (+ 5749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:11.638355 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:11.638371 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:11.638941 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:11.640749 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:11.640754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:11.640758 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:11.641030 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:11.642411 (+ 1381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:11.642415 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:11.642880 (+ 465us) client_negotiation.cc:769] Sending connection context
0810 14:32:11.643174 (+ 294us) client_negotiation.cc:240] Negotiation successful
0810 14:32:11.643706 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"mutex_wait_us":55,"spinlock_wait_cycles":18304,"thread_start_us":872,"threads_started":1}
I20260810 14:32:11.646133 18264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:11.611947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:38073 (local address 127.6.46.193:42747)
0810 14:32:11.618524 (+ 6577us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:11.618540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:11.618725 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:11.619537 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:11.619548 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:11.620567 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:11.621489 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:11.621516 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.624177 (+ 2661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.624183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:11.630588 (+ 6405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:11.630604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.632877 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.632883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:11.633049 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:11.633628 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:11.633650 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:11.636259 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:11.640387 (+ 4128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:11.640392 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:11.640395 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:11.641078 (+ 683us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:11.641492 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:11.641497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:11.641500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:11.641639 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:11.642383 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:11.642391 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:11.642862 (+ 471us) client_negotiation.cc:769] Sending connection context
0810 14:32:11.643175 (+ 313us) client_negotiation.cc:240] Negotiation successful
0810 14:32:11.645804 (+ 2629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6458,"mutex_wait_us":2279,"spinlock_wait_cycles":15232,"thread_start_us":6280,"threads_started":1}
Aug 10 14:32:11 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.46.194@KRBTEST.COM
I20260810 14:32:11.648744 18255 heartbeater.cc:347] Connected to a master server at 127.6.46.252:38073
I20260810 14:32:11.649019 18255 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:11.649694 18262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:11.610760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:37511 (local address 127.6.46.254:39189)
0810 14:32:11.611256 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:11.611263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:11.612293 (+ 1030us) server_negotiation.cc:407] Connection header received
0810 14:32:11.621742 (+ 9449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:11.621749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:11.621840 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:11.621959 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:11.623017 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.624485 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.626758 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.627430 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.627562 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:11.633685 (+ 6123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:11.633692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:11.633697 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:11.633739 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:11.638220 (+ 4481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:11.640466 (+ 2246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:11.640472 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:11.640477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:11.640646 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:11.641891 (+ 1245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:11.641896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:11.641900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:11.642304 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:11.642461 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:11.649008 (+ 6547us) server_negotiation.cc:299] Negotiation successful
0810 14:32:11.649278 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":220,"threads_started":1}
I20260810 14:32:11.649974 18255 heartbeater.cc:511] Master 127.6.46.252:38073 requested a full tablet report, sending...
Aug 10 14:32:11 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372331, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.253@KRBTEST.COM
I20260810 14:32:11.650869 18256 heartbeater.cc:347] Connected to a master server at 127.6.46.254:39189
I20260810 14:32:11.650996 18256 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:11.651700 18256 heartbeater.cc:511] Master 127.6.46.254:39189 requested a full tablet report, sending...
I20260810 14:32:11.652609 17950 ts_manager.cc:194] Registered new tserver with Master: a6e4c148a686452caad3c9f8cb9bf56c (127.6.46.193:36107)
I20260810 14:32:11.652923 16580 ts_manager.cc:194] Re-registered known tserver with Master: a6e4c148a686452caad3c9f8cb9bf56c (127.6.46.193:36107)
I20260810 14:32:11.654464 16580 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:37511
I20260810 14:32:11.659019 18273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:11.622154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:35217 (local address 127.6.46.193:58319)
0810 14:32:11.635200 (+ 13046us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:11.635228 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:11.635343 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:11.637169 (+ 1826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:11.637174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:11.637744 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:11.638198 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:11.638232 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.640562 (+ 2330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.640567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:11.646432 (+ 5865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:11.646447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.647422 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.647497 (+ 75us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:11.647693 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:11.648282 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:11.648302 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:11.653153 (+ 4851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:11.656667 (+ 3514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:11.656672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:11.656677 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:11.657065 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:11.657453 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:11.657457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:11.657461 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:11.657654 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:11.658203 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:11.658206 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:11.658350 (+ 144us) client_negotiation.cc:769] Sending connection context
0810 14:32:11.658486 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 14:32:11.658767 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12925,"thread_start_us":11314,"threads_started":1}
I20260810 14:32:11.659238 18269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:11.618818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:58319 (local address 127.6.46.253:35217)
0810 14:32:11.636664 (+ 17846us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:11.636670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:11.636690 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:32:11.636751 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:11.636756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:11.636843 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:11.636944 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:11.639262 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.640388 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.646591 (+ 6203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.647227 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.647342 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:11.653380 (+ 6038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:11.653388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:11.653392 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:11.653433 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:11.656538 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:11.657189 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:11.657194 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:11.657197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:11.657322 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:11.657779 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:11.657783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:11.657785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:11.658088 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:11.658255 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:11.658642 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 14:32:11.658863 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17680,"thread_start_us":16871,"threads_started":1}
I20260810 14:32:11.660065 18257 heartbeater.cc:347] Connected to a master server at 127.6.46.253:35217
I20260810 14:32:11.660180 18257 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:11.660498 18257 heartbeater.cc:511] Master 127.6.46.253:35217 requested a full tablet report, sending...
I20260810 14:32:11.660913 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:11.615739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33570 (local address 127.6.46.194:38723)
0810 14:32:11.629082 (+ 13343us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:11.629091 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:11.629132 (+ 41us) server_negotiation.cc:407] Connection header received
0810 14:32:11.629203 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:11.629212 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:11.629446 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:11.631895 (+ 2449us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:11.632180 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.638066 (+ 5886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.640833 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:11.641737 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:11.651036 (+ 9299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:11.651352 (+ 316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:11.651363 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:11.651380 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:11.651414 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:11.657295 (+ 5881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:11.658078 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:11.658087 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:11.658090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:11.658265 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:11.658689 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:11.658694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:11.658697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:11.659104 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:11.660253 (+ 1149us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:11.660386 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 14:32:11.660585 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13131,"thread_start_us":13843,"threads_started":1}
I20260810 14:32:11.662113 17619 ts_manager.cc:194] Registered new tserver with Master: a6e4c148a686452caad3c9f8cb9bf56c (127.6.46.193:36107)
I20260810 14:32:11.662220 17129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:11.745411 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 16910
I20260810 14:32:11.764707 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:38723
--local_ip_for_outbound_sockets=127.6.46.194
--tserver_master_addrs=127.6.46.254:39189
--webserver_port=45173
--webserver_interface=127.6.46.194
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab
--principal=oryx/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.46.254:39189,127.6.46.253:35217
--tserver_master_addrs=127.6.46.254:39189,127.6.46.253:35217,127.6.46.252:38073 with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:12.080868 18275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:12.081105 18275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:12.081205 18275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:12.088116 18275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:12.088208 18275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:12.088320 18275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:32:12.097904 18275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab
--principal=oryx/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:38723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=45173
--tserver_master_addrs=127.6.46.254:39189,127.6.46.253:35217,127.6.46.252:38073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:12.099164 18275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:12.100406 18275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:12.113029 18284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:12.112998 18285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:12.112948 18292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:12.114037 18275 server_base.cc:1034] running on GCE node
I20260810 14:32:12.114674 18275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:12.115532 18275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:12.116724 18275 hybrid_clock.cc:648] HybridClock initialized: now 1786372332116667 us; error 45 us; skew 500 ppm
I20260810 14:32:12.117009 18275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:12 dist-test-slave-q9h9 krb5kdc[16515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372332, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:12.119935 18275 init.cc:377] Logged in from keytab as oryx/127.6.46.194@KRBTEST.COM (short username oryx)
I20260810 14:32:12.121315 18275 webserver.cc:460] Webserver started at http://127.6.46.194:45173/ using document root <none> and password file <none>
I20260810 14:32:12.121930 18275 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:12.122073 18275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:12.127276 18275 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 14:32:12.129797 18302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:12.131032 18275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:32:12.131193 18275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "7358de53372848ca93f3d28c6e674f2d"
format_stamp: "Formatted at 2026-08-10 14:32:08 on dist-test-slave-q9h9"
server_key: "18dbb1b5d588d6bf3d742c350968833d"
server_key_iv: "da7fa3022036aa232e2b8453e1b7f519"
server_key_version: "encryptionkey@0"
I20260810 14:32:12.132104 18275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:12.154953 18275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:12.158712 18275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:12.158949 18275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:12.159930 18275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:12.161262 18275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:12.161360 18275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:12.161459 18275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:12.161530 18275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:12.192637 18275 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:38723
I20260810 14:32:12.192693 18486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:38723 every 8 connection(s)
I20260810 14:32:12.194044 18275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:32:12.196194 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 18275
I20260810 14:32:12.196456 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 17148
Aug 10 14:32:12 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.46.195@KRBTEST.COM
Aug 10 14:32:12 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372332, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.253@KRBTEST.COM
Aug 10 14:32:12 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372332, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
Aug 10 14:32:12 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372332, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.252@KRBTEST.COM
I20260810 14:32:12.235443 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.197809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.6.46.195:44219)
0810 14:32:12.208830 (+ 11021us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:12.208838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:12.209456 (+ 618us) server_negotiation.cc:407] Connection header received
0810 14:32:12.209828 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:12.209837 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:12.210102 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:12.210283 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:12.214247 (+ 3964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.215848 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.217687 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.218609 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.218734 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.225692 (+ 6958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:12.225707 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:12.225725 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:12.225778 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:12.229527 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.230797 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.230806 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.230810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.230988 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.231417 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.231422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.231426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.233066 (+ 1640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:12.233358 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:12.234741 (+ 1383us) server_negotiation.cc:299] Negotiation successful
0810 14:32:12.235015 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10317,"thread_start_us":10146,"threads_started":1}
I20260810 14:32:12.237062 17356 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:12.241825 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.196832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:36887 (local address 127.6.46.254:39189)
0810 14:32:12.212806 (+ 15974us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:12.212813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:12.221874 (+ 9061us) server_negotiation.cc:407] Connection header received
0810 14:32:12.222093 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:12.222098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:12.222190 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:12.222297 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:12.223868 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.224953 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.227741 (+ 2788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.228396 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.228477 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.233657 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:12.233663 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:12.233668 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:12.233709 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:12.237649 (+ 3940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.238954 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.238958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.238961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.239129 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.239673 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.239679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.239682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.240033 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:12.240158 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:12.241163 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0810 14:32:12.241407 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15828,"thread_start_us":15974,"threads_started":1}
I20260810 14:32:12.242000 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.198103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:46375 (local address 127.6.46.253:35217)
0810 14:32:12.220890 (+ 22787us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:12.220896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:12.220918 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:32:12.220985 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:12.220991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:12.221078 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:12.221194 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:12.224159 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.225236 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.227742 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.228349 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.228451 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.233358 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:12.233366 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:12.233371 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:12.233412 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:12.236892 (+ 3480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.238824 (+ 1932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.238829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.238833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.238988 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.240410 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.240415 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.240418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.240799 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:12.240932 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:12.241432 (+ 500us) server_negotiation.cc:299] Negotiation successful
0810 14:32:12.241647 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22653,"thread_start_us":24688,"threads_started":1}
I20260810 14:32:12.242743 18503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.198154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.194:36887)
0810 14:32:12.221769 (+ 23615us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:12.221789 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:12.221931 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:12.222985 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:12.222988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:12.223428 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:12.223661 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.223670 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.225162 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.225167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.227566 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.227581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.228818 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.228826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.229184 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.230102 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:12.230124 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:12.233436 (+ 3312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:12.237875 (+ 4439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:12.237883 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:12.237898 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:12.238801 (+ 903us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:12.239303 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:12.239307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:12.239328 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:12.239477 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:12.240304 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:12.240311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:12.240714 (+ 403us) client_negotiation.cc:769] Sending connection context
0810 14:32:12.241016 (+ 302us) client_negotiation.cc:240] Negotiation successful
0810 14:32:12.241492 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23479,"mutex_wait_us":209,"spinlock_wait_cycles":3200,"thread_start_us":24161,"threads_started":1}
I20260810 14:32:12.243990 18495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.197507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:35217 (local address 127.6.46.194:46375)
0810 14:32:12.208335 (+ 10828us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:12.208400 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:12.217023 (+ 8623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:12.221474 (+ 4451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:12.221486 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:12.222405 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:12.223319 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.223350 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.225405 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.225411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.227566 (+ 2155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.227581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.228818 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.228826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.229189 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.229988 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:12.230042 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:12.233074 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:12.237886 (+ 4812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:12.237891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:12.237898 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:12.238623 (+ 725us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:12.239141 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:12.239145 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:12.239149 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:12.240288 (+ 1139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:12.241013 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:12.241017 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:12.241171 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 14:32:12.243308 (+ 2137us) client_negotiation.cc:240] Negotiation successful
0810 14:32:12.243574 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9890,"mutex_wait_us":204,"spinlock_wait_cycles":28800,"thread_start_us":9678,"threads_started":1}
I20260810 14:32:12.244246 18489 heartbeater.cc:347] Connected to a master server at 127.6.46.254:39189
I20260810 14:32:12.244570 18489 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:12.244889 18490 heartbeater.cc:347] Connected to a master server at 127.6.46.253:35217
I20260810 14:32:12.245025 18490 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:12.245527 18490 heartbeater.cc:511] Master 127.6.46.253:35217 requested a full tablet report, sending...
I20260810 14:32:12.245522 18489 heartbeater.cc:511] Master 127.6.46.254:39189 requested a full tablet report, sending...
I20260810 14:32:12.247135 16580 ts_manager.cc:194] Re-registered known tserver with Master: 7358de53372848ca93f3d28c6e674f2d (127.6.46.194:38723)
I20260810 14:32:12.247309 17619 ts_manager.cc:194] Registered new tserver with Master: 7358de53372848ca93f3d28c6e674f2d (127.6.46.194:38723)
I20260810 14:32:12.248235 16580 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:36887
I20260810 14:32:12.250685 18504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.209314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:35207 (local address 127.6.46.252:38073)
0810 14:32:12.226178 (+ 16864us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:12.226184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:12.226205 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:32:12.226264 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:12.226269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:12.226351 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:12.227855 (+ 1504us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:12.228039 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.229142 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.231131 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.231726 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.231848 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.237117 (+ 5269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:12.237124 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:12.237128 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:12.237167 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:12.245869 (+ 8702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.246619 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.246624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.246628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.248581 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.249127 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.249132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.249136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.249417 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:12.249551 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:12.250079 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 14:32:12.250303 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16728,"thread_start_us":17128,"threads_started":1}
I20260810 14:32:12.250823 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.209299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:38073 (local address 127.6.46.194:35207)
0810 14:32:12.222584 (+ 13285us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:12.222597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:12.222764 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:12.226626 (+ 3862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:12.226631 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:12.227300 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:12.227698 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.227713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.229305 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.229311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.230943 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.230958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.231889 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.231893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.232026 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.232555 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:12.232577 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:12.235222 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:12.246038 (+ 10816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:12.246043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:12.246047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:12.246481 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:12.248735 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:12.248740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:12.248744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:12.249001 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:12.249535 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:12.249540 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:12.249755 (+ 215us) client_negotiation.cc:769] Sending connection context
0810 14:32:12.249916 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 14:32:12.250292 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13164,"mutex_wait_us":26,"spinlock_wait_cycles":1920,"thread_start_us":12726,"threads_started":1}
I20260810 14:32:12.251586 18488 heartbeater.cc:347] Connected to a master server at 127.6.46.252:38073
I20260810 14:32:12.251720 18488 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:12.252069 18488 heartbeater.cc:511] Master 127.6.46.252:38073 requested a full tablet report, sending...
I20260810 14:32:12.253116 17950 ts_manager.cc:194] Registered new tserver with Master: 7358de53372848ca93f3d28c6e674f2d (127.6.46.194:38723)
I20260810 14:32:12.326930 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 17148
I20260810 14:32:12.352385 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:44219
--local_ip_for_outbound_sockets=127.6.46.195
--tserver_master_addrs=127.6.46.254:39189
--webserver_port=44247
--webserver_interface=127.6.46.195
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab
--principal=oryx/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.46.254:39189,127.6.46.253:35217
--tserver_master_addrs=127.6.46.254:39189,127.6.46.253:35217,127.6.46.252:38073 with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:32:12.657509 18256 heartbeater.cc:503] Master 127.6.46.254:39189 was elected leader, sending a full tablet report...
W20260810 14:32:12.670931 18506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:12.671212 18506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:12.671345 18506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:12.678390 18506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:12.678495 18506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:12.678637 18506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:32:12.687815 18506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab
--principal=oryx/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:44219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=44247
--tserver_master_addrs=127.6.46.254:39189,127.6.46.253:35217,127.6.46.252:38073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:12.689073 18506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:12.690316 18506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:12.702562 18516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:12.702533 18517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:12.702497 18519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:12.703547 18506 server_base.cc:1034] running on GCE node
I20260810 14:32:12.704135 18506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:12.704959 18506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:12.706146 18506 hybrid_clock.cc:648] HybridClock initialized: now 1786372332706099 us; error 42 us; skew 500 ppm
I20260810 14:32:12.706416 18506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:12 dist-test-slave-q9h9 krb5kdc[16515](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372332, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:12.709360 18506 init.cc:377] Logged in from keytab as oryx/127.6.46.195@KRBTEST.COM (short username oryx)
I20260810 14:32:12.710727 18506 webserver.cc:460] Webserver started at http://127.6.46.195:44247/ using document root <none> and password file <none>
I20260810 14:32:12.711345 18506 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:12.711455 18506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:12.716498 18506 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 14:32:12.718954 18530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:12.720155 18506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:32:12.720283 18506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "031348ed4bcd474687b9a337f7cb4378"
format_stamp: "Formatted at 2026-08-10 14:32:09 on dist-test-slave-q9h9"
server_key: "b52bf6fee48da89c88f143a05b9eed73"
server_key_iv: "4a6aa59d911d45c7f84f8410de14d4b4"
server_key_version: "encryptionkey@0"
I20260810 14:32:12.721138 18506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:12.741448 18506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:12.744931 18506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:12.745151 18506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:12.745951 18506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:12.747195 18506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:12.747280 18506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:12.747356 18506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:12.747408 18506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:12.777074 18506 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:44219
I20260810 14:32:12.777161 18698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:44219 every 8 connection(s)
I20260810 14:32:12.778405 18506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:32:12.783752 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 18506
Aug 10 14:32:12 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372332, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.253@KRBTEST.COM
Aug 10 14:32:12 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372332, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
I20260810 14:32:12.828047 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.780962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:57191 (local address 127.6.46.254:39189)
0810 14:32:12.798143 (+ 17181us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:12.798150 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:12.798169 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:32:12.798238 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:12.798244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:12.798328 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:12.798434 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:12.806335 (+ 7901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.807851 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.811822 (+ 3971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.812422 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.812532 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.818067 (+ 5535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:12.818073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:12.818078 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:12.818116 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:12.822553 (+ 4437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.824058 (+ 1505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.824063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.824066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.824204 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.825710 (+ 1506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.825716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.825719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.826085 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:12.826222 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:12.827466 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0810 14:32:12.827695 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1230,"server-negotiator.queue_time_us":17036,"thread_start_us":12941,"threads_started":1}
I20260810 14:32:12.829046 18725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.797030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41956 (local address 127.6.46.254:39189)
0810 14:32:12.803067 (+ 6037us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:12.803074 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:12.803926 (+ 852us) server_negotiation.cc:407] Connection header received
0810 14:32:12.803990 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:12.803996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:12.804082 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:12.804180 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:12.806607 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.807968 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.809879 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.810610 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.810724 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.814506 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:12.814513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:12.814517 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:12.814553 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:12.822322 (+ 7769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.823436 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.823442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.823445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.823605 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.824334 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.824338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.824342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.824638 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:12.828358 (+ 3720us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:12.828478 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 14:32:12.828694 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5890,"thread_start_us":5769,"threads_started":1}
I20260810 14:32:12.832095 18711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.781608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.195:57191)
0810 14:32:12.794587 (+ 12979us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:12.794638 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:12.797079 (+ 2441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:12.805373 (+ 8294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:12.805378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:12.805867 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:12.806164 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.806176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.808054 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.808059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.811692 (+ 3633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.811708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.814266 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.814273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.814479 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.815069 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:12.815089 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:12.817768 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:12.823238 (+ 5470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:12.823246 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:12.823264 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:12.823915 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:12.824669 (+ 754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:12.824674 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:12.824677 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:12.824923 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:12.826254 (+ 1331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:12.826262 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:12.826729 (+ 467us) client_negotiation.cc:769] Sending connection context
0810 14:32:12.827023 (+ 294us) client_negotiation.cc:240] Negotiation successful
0810 14:32:12.830662 (+ 3639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12630,"thread_start_us":12386,"threads_started":1}
I20260810 14:32:12.833808 18703 heartbeater.cc:347] Connected to a master server at 127.6.46.254:39189
I20260810 14:32:12.834153 18703 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:12.834347 18710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.781048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:43229 (local address 127.6.46.253:35217)
0810 14:32:12.793361 (+ 12313us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:12.793368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:12.797845 (+ 4477us) server_negotiation.cc:407] Connection header received
0810 14:32:12.798805 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:12.798812 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:12.798905 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:12.798981 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:12.801550 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.802764 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.809321 (+ 6557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.809964 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.810083 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.819150 (+ 9067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:12.819157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:12.819162 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:12.819217 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:12.823096 (+ 3879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.825080 (+ 1984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.825086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.825089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.825238 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.825684 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.825688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.825692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.826049 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:12.826174 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:12.833741 (+ 7567us) server_negotiation.cc:299] Negotiation successful
0810 14:32:12.833964 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12175,"thread_start_us":12553,"threads_started":1}
I20260810 14:32:12.835013 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.797125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.6.46.253:35217)
0810 14:32:12.815639 (+ 18514us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:12.815645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:12.815664 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:32:12.815714 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:12.815719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:12.815793 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:12.815897 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:12.817960 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.819179 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.820982 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.821650 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.821762 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.826609 (+ 4847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:12.826616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:12.826620 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:12.826656 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:12.830167 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.831431 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.831436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.831439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.831781 (+ 342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.832256 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.832262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.832265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.832630 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:12.832844 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:12.833789 (+ 945us) server_negotiation.cc:299] Negotiation successful
0810 14:32:12.834647 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":659,"server-negotiator.queue_time_us":18390,"thread_start_us":18254,"threads_started":1}
I20260810 14:32:12.835044 18703 heartbeater.cc:511] Master 127.6.46.254:39189 requested a full tablet report, sending...
I20260810 14:32:12.836289 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.781608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:35217 (local address 127.6.46.195:43229)
0810 14:32:12.797740 (+ 16132us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:12.797763 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:12.798640 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:12.799462 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:12.799474 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:12.800377 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:12.801199 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.801224 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.802976 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.802982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.805211 (+ 2229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.805227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.810484 (+ 5257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.810492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.810768 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.811974 (+ 1206us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:12.812004 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:12.818354 (+ 6350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:12.824075 (+ 5721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:12.824079 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:12.824082 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:12.824543 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:12.825382 (+ 839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:12.825386 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:12.825389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:12.825541 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:12.830950 (+ 5409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:12.830955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:12.831131 (+ 176us) client_negotiation.cc:769] Sending connection context
0810 14:32:12.834516 (+ 3385us) client_negotiation.cc:240] Negotiation successful
0810 14:32:12.834813 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15983,"mutex_wait_us":33,"spinlock_wait_cycles":26368,"thread_start_us":20297,"threads_started":1}
I20260810 14:32:12.837152 18704 heartbeater.cc:347] Connected to a master server at 127.6.46.253:35217
I20260810 14:32:12.837107 16578 ts_manager.cc:194] Re-registered known tserver with Master: 031348ed4bcd474687b9a337f7cb4378 (127.6.46.195:44219)
I20260810 14:32:12.837282 18704 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:12.837700 18704 heartbeater.cc:511] Master 127.6.46.253:35217 requested a full tablet report, sending...
I20260810 14:32:12.838373 16578 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:57191
I20260810 14:32:12.838820 17619 ts_manager.cc:194] Registered new tserver with Master: 031348ed4bcd474687b9a337f7cb4378 (127.6.46.195:44219)
Aug 10 14:32:12 dist-test-slave-q9h9 krb5kdc[16515](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372332, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.252@KRBTEST.COM
I20260810 14:32:12.833832 16580 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 14:32:12.843492 18733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.815134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.6.46.252:38073)
0810 14:32:12.822854 (+ 7720us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:12.822860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:12.823787 (+ 927us) server_negotiation.cc:407] Connection header received
0810 14:32:12.823846 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:12.823851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:12.823934 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:12.824088 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:12.826341 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.827466 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.829242 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.829865 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.829975 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.835264 (+ 5289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:12.835270 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:12.835274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:12.835311 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:12.840462 (+ 5151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.841278 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.841284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.841287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.841460 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.841898 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.841905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.841907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.842297 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:12.842562 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:12.842885 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 14:32:12.843127 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7593,"thread_start_us":7808,"threads_started":1}
W20260810 14:32:12.844092 16580 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:32:12.847781 18731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.802579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:56625 (local address 127.6.46.252:38073)
0810 14:32:12.818618 (+ 16039us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:12.818624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:12.818642 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:32:12.818704 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:12.818709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:12.818782 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:12.818882 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:12.822542 (+ 3660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.824744 (+ 2202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.827940 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.828915 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.829046 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.841814 (+ 12768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:12.841821 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:12.841826 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:12.841860 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:12.845345 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.845919 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.845924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.845927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.846153 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:12.846535 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:12.846540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:12.846543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:12.846790 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:12.846897 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:12.847297 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 14:32:12.847443 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15898,"thread_start_us":12371,"threads_started":1}
I20260810 14:32:12.848639 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.802478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:38073 (local address 127.6.46.195:56625)
0810 14:32:12.811293 (+ 8815us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:12.811310 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:12.811410 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:12.821250 (+ 9840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:12.821255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:12.821801 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:12.822133 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.822146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.826296 (+ 4150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.826301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.827779 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.827793 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.829427 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.829433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.829593 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.830139 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:12.830160 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:12.841600 (+ 11440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:12.845465 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:12.845468 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:12.845471 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:12.845785 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:12.846248 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:12.846251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:12.846254 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:12.846398 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:12.846921 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:12.846926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:12.847082 (+ 156us) client_negotiation.cc:769] Sending connection context
0810 14:32:12.847222 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 14:32:12.847507 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8695,"thread_start_us":8536,"threads_started":1}
I20260810 14:32:12.849478 18702 heartbeater.cc:347] Connected to a master server at 127.6.46.252:38073
I20260810 14:32:12.849612 18702 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:12.849988 18702 heartbeater.cc:511] Master 127.6.46.252:38073 requested a full tablet report, sending...
I20260810 14:32:12.852535 17950 ts_manager.cc:194] Registered new tserver with Master: 031348ed4bcd474687b9a337f7cb4378 (127.6.46.195:44219)
I20260810 14:32:12.876313 18752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.865922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:44219 (local address 127.0.0.1:41538)
0810 14:32:12.867131 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:12.867150 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:12.867284 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:12.868460 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:12.868465 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:12.868491 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:12.869948 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.869965 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.871915 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.871921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.873968 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.873980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.875385 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.875391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.875586 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.875692 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 14:32:12.875782 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 14:32:12.875901 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"mutex_wait_us":38,"thread_start_us":905,"threads_started":1}
I20260810 14:32:12.876348 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.866201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.6.46.195:44219)
0810 14:32:12.867507 (+ 1306us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:12.867517 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:12.867551 (+ 34us) server_negotiation.cc:407] Connection header received
0810 14:32:12.867654 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:12.867663 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:12.867931 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:12.868148 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:12.870262 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.871770 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.874144 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.875172 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.875294 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.875496 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:12.875860 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 14:32:12.876023 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":261,"threads_started":1}
I20260810 14:32:12.878962 18756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.866839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:36107 (local address 127.0.0.1:43684)
0810 14:32:12.870594 (+ 3755us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:12.870611 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:12.870845 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:12.871486 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:12.871491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:12.871527 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:12.871780 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.871789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.873669 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.873674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.875683 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.875696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.878181 (+ 2485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.878188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.878348 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.878382 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 14:32:12.878473 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 14:32:12.878597 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3622,"mutex_wait_us":3,"thread_start_us":4193,"threads_started":1}
I20260810 14:32:12.878973 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.867062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.6.46.193:36107)
0810 14:32:12.869894 (+ 2832us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:12.869903 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:12.870837 (+ 934us) server_negotiation.cc:407] Connection header received
0810 14:32:12.870995 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:12.871012 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:12.871242 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:12.871430 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:12.872084 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.873526 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.876615 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.877960 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.878118 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.878314 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:12.878534 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 14:32:12.878683 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2649,"thread_start_us":2507,"threads_started":1}
I20260810 14:32:12.880249 18757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.867672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33582 (local address 127.6.46.194:38723)
0810 14:32:12.869313 (+ 1641us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:12.869322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:12.871403 (+ 2081us) server_negotiation.cc:407] Connection header received
0810 14:32:12.872162 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:12.872171 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:12.872425 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:12.872658 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:12.873440 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.874862 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.877254 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.879361 (+ 2107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.879476 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.879662 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:12.879923 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 14:32:12.880054 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1480,"thread_start_us":1373,"threads_started":1}
I20260810 14:32:12.880338 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:12.867188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:38723 (local address 127.0.0.1:33582)
0810 14:32:12.871285 (+ 4097us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:12.871301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:12.871418 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:12.872935 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:12.872939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:12.872957 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:12.873301 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.873309 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.875006 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.875012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.877084 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:12.877100 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:12.879579 (+ 2479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:12.879586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:12.879762 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:12.879793 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 14:32:12.879888 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 14:32:12.880000 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3976,"mutex_wait_us":32,"thread_start_us":3126,"threads_started":1}
I20260810 14:32:12.880328 18597 tablet_service.cc:1467] Processing CreateTablet for tablet d545b569d4f04c088c1c14d2a5883b70 (DEFAULT_TABLE table=test-table [id=b53333d0d8de4acd8423c1091dd6dfee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:12.881745 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d545b569d4f04c088c1c14d2a5883b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:12.883409 18383 tablet_service.cc:1467] Processing CreateTablet for tablet d545b569d4f04c088c1c14d2a5883b70 (DEFAULT_TABLE table=test-table [id=b53333d0d8de4acd8423c1091dd6dfee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:12.884471 18383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d545b569d4f04c088c1c14d2a5883b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:12.884734 18133 tablet_service.cc:1467] Processing CreateTablet for tablet d545b569d4f04c088c1c14d2a5883b70 (DEFAULT_TABLE table=test-table [id=b53333d0d8de4acd8423c1091dd6dfee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:12.885768 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d545b569d4f04c088c1c14d2a5883b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:12.897863 18759 tablet_bootstrap.cc:492] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d: Bootstrap starting.
I20260810 14:32:12.901571 18759 tablet_bootstrap.cc:654] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:12.902940 18761 tablet_bootstrap.cc:492] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c: Bootstrap starting.
I20260810 14:32:12.902946 18759 log.cc:824] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:12.906854 18761 tablet_bootstrap.cc:654] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:12.908177 18761 log.cc:824] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:12.912046 18763 tablet_bootstrap.cc:492] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378: Bootstrap starting.
I20260810 14:32:12.913961 18759 tablet_bootstrap.cc:492] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d: No bootstrap required, opened a new log
I20260810 14:32:12.914256 18759 ts_tablet_manager.cc:1397] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260810 14:32:12.920362 18761 tablet_bootstrap.cc:492] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c: No bootstrap required, opened a new log
I20260810 14:32:12.920637 18761 ts_tablet_manager.cc:1397] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.009s
I20260810 14:32:12.923501 18763 tablet_bootstrap.cc:654] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:12.924817 18763 log.cc:824] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:12.925143 18759 raft_consensus.cc:348] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } }
I20260810 14:32:12.925625 18759 raft_consensus.cc:374] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:12.925737 18759 raft_consensus.cc:729] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7358de53372848ca93f3d28c6e674f2d, State: Initialized, Role: FOLLOWER
I20260810 14:32:12.926364 18759 consensus_queue.cc:260] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } }
I20260810 14:32:12.928113 18763 tablet_bootstrap.cc:492] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378: No bootstrap required, opened a new log
I20260810 14:32:12.928424 18763 ts_tablet_manager.cc:1397] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.007s
I20260810 14:32:12.931347 18759 ts_tablet_manager.cc:1428] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d: Time spent starting tablet: real 0.017s user 0.010s sys 0.005s
I20260810 14:32:12.931226 18761 raft_consensus.cc:348] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } }
I20260810 14:32:12.931617 18489 heartbeater.cc:503] Master 127.6.46.254:39189 was elected leader, sending a full tablet report...
I20260810 14:32:12.931619 18761 raft_consensus.cc:374] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:12.931722 18761 raft_consensus.cc:729] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e4c148a686452caad3c9f8cb9bf56c, State: Initialized, Role: FOLLOWER
I20260810 14:32:12.932279 18761 consensus_queue.cc:260] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } }
I20260810 14:32:12.937520 18761 ts_tablet_manager.cc:1428] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c: Time spent starting tablet: real 0.017s user 0.002s sys 0.017s
I20260810 14:32:12.939109 18763 raft_consensus.cc:348] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } }
I20260810 14:32:12.939515 18763 raft_consensus.cc:374] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:12.939607 18763 raft_consensus.cc:729] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031348ed4bcd474687b9a337f7cb4378, State: Initialized, Role: FOLLOWER
I20260810 14:32:12.940261 18763 consensus_queue.cc:260] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } }
I20260810 14:32:12.941502 18703 heartbeater.cc:503] Master 127.6.46.254:39189 was elected leader, sending a full tablet report...
I20260810 14:32:12.941915 18763 ts_tablet_manager.cc:1428] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
W20260810 14:32:12.947923 18492 tablet.cc:2369] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:32:13.018615 18768 raft_consensus.cc:482] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:32:13.018909 18768 raft_consensus.cc:504] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } }
I20260810 14:32:13.020514 18768 leader_election.cc:288] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031348ed4bcd474687b9a337f7cb4378 (127.6.46.195:44219), 7358de53372848ca93f3d28c6e674f2d (127.6.46.194:38723)
I20260810 14:32:13.027860 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:13.021004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:38693 (local address 127.6.46.195:44219)
0810 14:32:13.021239 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:13.021246 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:13.021394 (+ 148us) server_negotiation.cc:407] Connection header received
0810 14:32:13.021599 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:13.021604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:13.021681 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:13.021767 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:13.022575 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.023502 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.025932 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.027098 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.027271 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:13.027348 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:13.027571 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 14:32:13.027684 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:32:13.028051 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:13.020712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:44219 (local address 127.6.46.193:38693)
0810 14:32:13.021298 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:13.021313 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:13.021466 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:13.021972 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:13.021978 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:13.022007 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:13.022447 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:13.022459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.023623 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.023629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:13.025792 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:13.025807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.027296 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.027300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:13.027414 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:13.027437 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:32:13.027534 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 14:32:13.027662 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"mutex_wait_us":43,"thread_start_us":275,"threads_started":1}
I20260810 14:32:13.028283 18757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:13.021684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:53953 (local address 127.6.46.194:38723)
0810 14:32:13.021892 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:13.021898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:13.022288 (+ 390us) server_negotiation.cc:407] Connection header received
0810 14:32:13.022469 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:13.022473 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:13.022533 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:13.022637 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:13.023137 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.023907 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.026190 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.027398 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.027511 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:13.027577 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:13.027917 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 14:32:13.028043 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 14:32:13.028327 18772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:13.021418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:38723 (local address 127.6.46.193:53953)
0810 14:32:13.022195 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:13.022215 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:13.022345 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:13.022655 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:13.022659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:13.022684 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:13.022975 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:13.022986 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.024042 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.024048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:13.026044 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:13.026055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.027590 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.027597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:13.027752 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:13.027789 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 14:32:13.027874 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 14:32:13.027988 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":1,"thread_start_us":433,"threads_started":1}
I20260810 14:32:13.029222 18630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d545b569d4f04c088c1c14d2a5883b70" candidate_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031348ed4bcd474687b9a337f7cb4378" is_pre_election: true
I20260810 14:32:13.029676 18630 raft_consensus.cc:2454] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e4c148a686452caad3c9f8cb9bf56c in term 0.
I20260810 14:32:13.029462 18413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d545b569d4f04c088c1c14d2a5883b70" candidate_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7358de53372848ca93f3d28c6e674f2d" is_pre_election: true
I20260810 14:32:13.030198 18413 raft_consensus.cc:2454] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6e4c148a686452caad3c9f8cb9bf56c in term 0.
I20260810 14:32:13.030447 18064 leader_election.cc:302] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031348ed4bcd474687b9a337f7cb4378, a6e4c148a686452caad3c9f8cb9bf56c; no voters:
I20260810 14:32:13.030912 18768 raft_consensus.cc:2781] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260810 14:32:13.031076 18706 tablet.cc:2369] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:32:13.031126 18768 raft_consensus.cc:482] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:32:13.031239 18768 raft_consensus.cc:3037] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:13.033372 18768 raft_consensus.cc:504] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } }
I20260810 14:32:13.033941 18768 leader_election.cc:288] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [CANDIDATE]: Term 1 election: Requested vote from peers 031348ed4bcd474687b9a337f7cb4378 (127.6.46.195:44219), 7358de53372848ca93f3d28c6e674f2d (127.6.46.194:38723)
I20260810 14:32:13.034300 18630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d545b569d4f04c088c1c14d2a5883b70" candidate_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031348ed4bcd474687b9a337f7cb4378"
I20260810 14:32:13.034395 18413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d545b569d4f04c088c1c14d2a5883b70" candidate_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7358de53372848ca93f3d28c6e674f2d"
I20260810 14:32:13.034466 18630 raft_consensus.cc:3037] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:13.034584 18413 raft_consensus.cc:3037] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:13.036700 18630 raft_consensus.cc:2454] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e4c148a686452caad3c9f8cb9bf56c in term 1.
I20260810 14:32:13.036972 18413 raft_consensus.cc:2454] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6e4c148a686452caad3c9f8cb9bf56c in term 1.
I20260810 14:32:13.037053 18064 leader_election.cc:302] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031348ed4bcd474687b9a337f7cb4378, a6e4c148a686452caad3c9f8cb9bf56c; no voters:
I20260810 14:32:13.037462 18768 raft_consensus.cc:2781] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:13.037886 18768 raft_consensus.cc:686] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [term 1 LEADER]: Becoming Leader. State: Replica: a6e4c148a686452caad3c9f8cb9bf56c, State: Running, Role: LEADER
I20260810 14:32:13.038668 18768 consensus_queue.cc:237] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } }
I20260810 14:32:13.043418 16580 catalog_manager.cc:5495] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c reported cstate change: term changed from 0 to 1, leader changed from <none> to a6e4c148a686452caad3c9f8cb9bf56c (127.6.46.193). New cstate: current_term: 1 leader_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } health_report { overall_health: HEALTHY } } }
I20260810 14:32:13.063999 18755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:13.058930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43686 (local address 127.6.46.193:36107)
0810 14:32:13.059093 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:13.059099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:13.059292 (+ 193us) server_negotiation.cc:407] Connection header received
0810 14:32:13.059476 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:13.059480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:13.059548 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:13.059669 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:32:13.060020 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.060722 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.062002 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.062452 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.062560 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:13.062998 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:32:13.063461 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:32:13.063599 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:13.063697 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 14:32:13.063770 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:32:13.071460 18630 raft_consensus.cc:1261] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378 [term 1 FOLLOWER]: Refusing update from remote peer a6e4c148a686452caad3c9f8cb9bf56c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:13.071529 18413 raft_consensus.cc:1261] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d [term 1 FOLLOWER]: Refusing update from remote peer a6e4c148a686452caad3c9f8cb9bf56c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:13.072327 18773 consensus_queue.cc:1035] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [LEADER]: Connected to new peer: Peer: permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:13.072659 18768 consensus_queue.cc:1035] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:13.107263 16580 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:13.107570 16580 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:13.114665 16580 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd97ad2506a34450b6cf70414423528c: Sending DeleteTablet for 3 replicas of tablet d545b569d4f04c088c1c14d2a5883b70
I20260810 14:32:13.115816 18133 tablet_service.cc:1514] Processing DeleteTablet for tablet d545b569d4f04c088c1c14d2a5883b70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:13 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:43684
I20260810 14:32:13.115902 18383 tablet_service.cc:1514] Processing DeleteTablet for tablet d545b569d4f04c088c1c14d2a5883b70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:13 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:33582
I20260810 14:32:13.116467 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet d545b569d4f04c088c1c14d2a5883b70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:13 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:41538
I20260810 14:32:13.116868 18790 tablet_replica.cc:330] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d: stopping tablet replica
I20260810 14:32:13.119180 18789 tablet_replica.cc:330] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c: stopping tablet replica
I20260810 14:32:13.119588 18789 raft_consensus.cc:2229] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:32:13.120282 18790 raft_consensus.cc:2229] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:13.120795 18790 raft_consensus.cc:2258] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:13.122268 18791 tablet_replica.cc:330] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378: stopping tablet replica
I20260810 14:32:13.122339 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 16536
I20260810 14:32:13.122705 18791 raft_consensus.cc:2229] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:13.123124 18789 raft_consensus.cc:2258] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:13.123157 18791 raft_consensus.cc:2258] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:13.123409 16640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:13.123862 18790 ts_tablet_manager.cc:1905] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:13.125321 18791 ts_tablet_manager.cc:1905] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:13.129668 18789 ts_tablet_manager.cc:1905] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:13.130556 18790 ts_tablet_manager.cc:1918] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:13.130717 18790 log.cc:1196] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/d545b569d4f04c088c1c14d2a5883b70
I20260810 14:32:13.131278 18790 ts_tablet_manager.cc:1939] T d545b569d4f04c088c1c14d2a5883b70 P 7358de53372848ca93f3d28c6e674f2d: Deleting consensus metadata
I20260810 14:32:13.131937 18791 ts_tablet_manager.cc:1918] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:13.132107 18791 log.cc:1196] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/d545b569d4f04c088c1c14d2a5883b70
I20260810 14:32:13.132640 18791 ts_tablet_manager.cc:1939] T d545b569d4f04c088c1c14d2a5883b70 P 031348ed4bcd474687b9a337f7cb4378: Deleting consensus metadata
I20260810 14:32:13.140393 18789 ts_tablet_manager.cc:1918] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:13.140578 18789 log.cc:1196] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/d545b569d4f04c088c1c14d2a5883b70
I20260810 14:32:13.141158 18789 ts_tablet_manager.cc:1939] T d545b569d4f04c088c1c14d2a5883b70 P a6e4c148a686452caad3c9f8cb9bf56c: Deleting consensus metadata
I20260810 14:32:13.148454 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:13.117824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.6.46.253:35217)
0810 14:32:13.118008 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:13.118045 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:13.122001 (+ 3956us) server_negotiation.cc:407] Connection header received
0810 14:32:13.122109 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:13.122114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:13.122190 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:13.122292 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:13.133533 (+ 11241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.134549 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.136469 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.137130 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.137254 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:13.141734 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:13.141742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:13.141748 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:13.141791 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:13.145326 (+ 3535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:13.146163 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:13.146169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:13.146173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:13.146327 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:13.146737 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:13.146743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:13.146747 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:13.147120 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:13.147387 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:13.147852 (+ 465us) server_negotiation.cc:299] Negotiation successful
0810 14:32:13.148101 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:32:13.164085 18731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:13.121903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58882 (local address 127.6.46.252:38073)
0810 14:32:13.122108 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:13.122114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:13.148399 (+ 26285us) server_negotiation.cc:407] Connection header received
0810 14:32:13.148971 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:13.148978 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:13.149078 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:13.149169 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:13.151437 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.152586 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.154880 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.155724 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.155832 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:13.159120 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:13.159125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:13.159128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:13.159160 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:13.161633 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:13.162283 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:13.162287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:13.162289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:13.162418 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:13.162772 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:13.162776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:13.162778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:13.163029 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:13.163278 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:13.163582 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 14:32:13.163798 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:32:13.331143 16560 catalog_manager.cc:4841] TS a6e4c148a686452caad3c9f8cb9bf56c (127.6.46.193:36107): tablet d545b569d4f04c088c1c14d2a5883b70 (table test-table [id=b53333d0d8de4acd8423c1091dd6dfee]) successfully deleted
I20260810 14:32:13.331773 16562 catalog_manager.cc:4841] TS 7358de53372848ca93f3d28c6e674f2d (127.6.46.194:38723): tablet d545b569d4f04c088c1c14d2a5883b70 (table test-table [id=b53333d0d8de4acd8423c1091dd6dfee]) successfully deleted
W20260810 14:32:13.332135 16562 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.46.194:38723, user_credentials={real_user=slave}} blocked reactor thread for 199001us
I20260810 14:32:13.333199 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 16536
I20260810 14:32:13.382009 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:13.365861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.6.46.253:35217)
0810 14:32:13.366092 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:13.366098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:13.367132 (+ 1034us) server_negotiation.cc:407] Connection header received
0810 14:32:13.367404 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:13.367408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:13.367479 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:13.367588 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:13.369179 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.369909 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.371725 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.372356 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.372479 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:13.375553 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:13.375559 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:13.375562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:13.375605 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:13.378762 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:13.379545 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:13.379551 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:13.379554 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:13.379705 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:13.380123 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:13.380128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:13.380132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:13.380456 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:13.380684 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:13.381428 (+ 744us) server_negotiation.cc:299] Negotiation successful
0810 14:32:13.381655 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:32:13.382308 18731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:13.367105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58896 (local address 127.6.46.252:38073)
0810 14:32:13.367285 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:13.367291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:13.367530 (+ 239us) server_negotiation.cc:407] Connection header received
0810 14:32:13.367741 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:13.367746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:13.367824 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:13.367891 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:13.370069 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.370768 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.372371 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:13.372988 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:13.373103 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:13.376101 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:13.376108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:13.376113 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:13.376155 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:13.379202 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:13.379915 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:13.379921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:13.379925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:13.380072 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:13.380439 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:13.380444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:13.380447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:13.380809 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:13.381056 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:13.381810 (+ 754us) server_negotiation.cc:299] Negotiation successful
0810 14:32:13.381966 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 14:32:14.091780 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.090902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.195:56383)
0810 14:32:14.091394 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.091543 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":261,"threads_started":1}
W20260810 14:32:14.092188 18703 heartbeater.cc:644] Failed to heartbeat to 127.6.46.254:39189 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
I20260810 14:32:14.100093 18818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.099381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.193:35691)
0810 14:32:14.099834 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.099941 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":255,"threads_started":1}
W20260810 14:32:14.100418 18256 heartbeater.cc:644] Failed to heartbeat to 127.6.46.254:39189 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
I20260810 14:32:14.102952 18819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.101985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.194:51673)
0810 14:32:14.102616 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.102754 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":411,"threads_started":1}
W20260810 14:32:14.103477 18489 heartbeater.cc:644] Failed to heartbeat to 127.6.46.254:39189 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
I20260810 14:32:14.614588 18820 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bd97ad2506a34450b6cf70414423528c)
I20260810 14:32:14.614888 18820 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } }
I20260810 14:32:14.615141 18821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bd97ad2506a34450b6cf70414423528c)
I20260810 14:32:14.615383 18821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } }
I20260810 14:32:14.616595 18820 leader_election.cc:288] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd97ad2506a34450b6cf70414423528c (127.6.46.254:39189), 5374e2ca086d4a858725cd7980b705d4 (127.6.46.253:35217)
I20260810 14:32:14.616751 18821 leader_election.cc:288] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd97ad2506a34450b6cf70414423528c (127.6.46.254:39189), f3092a8b44094e42b160704fc7b67d66 (127.6.46.252:38073)
I20260810 14:32:14.626933 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.616677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:42352)
0810 14:32:14.625755 (+ 9078us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.625931 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8958,"threads_started":1}
I20260810 14:32:14.627313 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.619956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:42368)
0810 14:32:14.626824 (+ 6868us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.627029 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6742,"thread_start_us":6557,"threads_started":1}
W20260810 14:32:14.627616 17590 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
W20260810 14:32:14.629359 17920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
I20260810 14:32:14.629635 18832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.629189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:42380)
0810 14:32:14.629401 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.629487 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260810 14:32:14.629904 17590 leader_election.cc:334] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bd97ad2506a34450b6cf70414423528c (127.6.46.254:39189): Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
I20260810 14:32:14.631594 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.631174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:42382)
0810 14:32:14.631337 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.631442 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 14:32:14.631858 17920 leader_election.cc:334] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer bd97ad2506a34450b6cf70414423528c (127.6.46.254:39189): Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
I20260810 14:32:14.634765 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.616759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:35217 (local address 127.0.0.1:38538)
0810 14:32:14.627073 (+ 10314us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.627091 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:14.627234 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:14.628010 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:14.628015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:14.628043 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:14.628414 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.628429 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.630337 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.630343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.632448 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.632462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.634108 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.634116 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.634254 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.634280 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 14:32:14.634365 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 14:32:14.634461 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10198,"mutex_wait_us":43,"thread_start_us":8738,"threads_started":1}
I20260810 14:32:14.634871 18827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.617169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.6.46.253:35217)
0810 14:32:14.627727 (+ 10558us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:14.627731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:14.627746 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:32:14.627799 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:14.627803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:14.627875 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:14.627972 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:14.628663 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.630182 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.632604 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.633883 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.634037 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.634160 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:14.634395 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 14:32:14.634529 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":10457,"thread_start_us":10278,"threads_started":1}
I20260810 14:32:14.635537 17631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f3092a8b44094e42b160704fc7b67d66" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5374e2ca086d4a858725cd7980b705d4" is_pre_election: true
I20260810 14:32:14.635914 17631 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3092a8b44094e42b160704fc7b67d66 in term 1.
I20260810 14:32:14.636468 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.617160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:38073 (local address 127.0.0.1:58912)
0810 14:32:14.626145 (+ 8985us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.626166 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:14.626310 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:14.629733 (+ 3423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:14.629738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:14.629763 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:14.630167 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.630179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.632168 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.632178 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.634162 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.634175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.635696 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.635705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.635886 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.635913 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 14:32:14.635984 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 14:32:14.636108 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8826,"mutex_wait_us":48,"thread_start_us":8684,"threads_started":1}
I20260810 14:32:14.636443 17930 leader_election.cc:302] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5374e2ca086d4a858725cd7980b705d4, f3092a8b44094e42b160704fc7b67d66; no voters: bd97ad2506a34450b6cf70414423528c
I20260810 14:32:14.636678 18834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.627322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58912 (local address 127.6.46.252:38073)
0810 14:32:14.629423 (+ 2101us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:14.629429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:14.629446 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:32:14.629502 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:14.629506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:14.629568 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:14.630333 (+ 765us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:14.630725 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.631959 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.634302 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.635490 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.635625 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.635723 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:14.636041 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 14:32:14.636178 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1961,"thread_start_us":1812,"threads_started":1}
I20260810 14:32:14.636843 18820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 14:32:14.636974 18820 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 FOLLOWER]: Starting leader election (detected failure of leader bd97ad2506a34450b6cf70414423528c)
I20260810 14:32:14.637087 18820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:32:14.637244 17965 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5374e2ca086d4a858725cd7980b705d4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f3092a8b44094e42b160704fc7b67d66" is_pre_election: true
I20260810 14:32:14.639253 18820 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } }
I20260810 14:32:14.639734 17965 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5374e2ca086d4a858725cd7980b705d4 in current term 2: Already voted for candidate f3092a8b44094e42b160704fc7b67d66 in this term.
I20260810 14:32:14.639886 18820 leader_election.cc:288] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [CANDIDATE]: Term 2 election: Requested vote from peers bd97ad2506a34450b6cf70414423528c (127.6.46.254:39189), 5374e2ca086d4a858725cd7980b705d4 (127.6.46.253:35217)
I20260810 14:32:14.640425 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.639948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:42388)
0810 14:32:14.640213 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.640297 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260810 14:32:14.640432 17631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f3092a8b44094e42b160704fc7b67d66" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5374e2ca086d4a858725cd7980b705d4"
I20260810 14:32:14.640671 17631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:32:14.640789 17594 leader_election.cc:302] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5374e2ca086d4a858725cd7980b705d4; no voters: bd97ad2506a34450b6cf70414423528c, f3092a8b44094e42b160704fc7b67d66
I20260810 14:32:14.642566 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.642159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:42406)
0810 14:32:14.642315 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.642405 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 14:32:14.642904 17920 leader_election.cc:334] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer bd97ad2506a34450b6cf70414423528c (127.6.46.254:39189): Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
I20260810 14:32:14.643438 17631 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3092a8b44094e42b160704fc7b67d66 in term 2.
I20260810 14:32:14.643625 18821 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:32:14.643795 17930 leader_election.cc:302] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5374e2ca086d4a858725cd7980b705d4, f3092a8b44094e42b160704fc7b67d66; no voters: bd97ad2506a34450b6cf70414423528c
I20260810 14:32:14.644040 18820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:32:14.644539 18820 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [term 2 LEADER]: Becoming Leader. State: Replica: f3092a8b44094e42b160704fc7b67d66, State: Running, Role: LEADER
I20260810 14:32:14.644972 18820 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } }
I20260810 14:32:14.646522 18838 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3092a8b44094e42b160704fc7b67d66. Latest consensus state: current_term: 2 leader_uuid: "f3092a8b44094e42b160704fc7b67d66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } } }
I20260810 14:32:14.646823 18838 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:14.647413 18842 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:32:14.651237 18842 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:32:14.652011 18842 catalog_manager.cc:1259] Loaded cluster ID: dfcf555871a444528a1ebb2de1eda106
I20260810 14:32:14.652091 18842 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:32:14.653203 18842 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:32:14.653998 18842 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: Loaded TSK: 0
I20260810 14:32:14.654821 18842 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:32:14.700269 17950 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:32:14.707221 17950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:32:14.711104 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.710740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:42422)
0810 14:32:14.710883 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.710972 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 14:32:14.711247 17631 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [term 2 FOLLOWER]: Refusing update from remote peer f3092a8b44094e42b160704fc7b67d66: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 14:32:14.711993 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.711640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:42434)
0810 14:32:14.711790 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.711856 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 14:32:14.712028 18838 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 14:32:14.712344 17920 consensus_peers.cc:487] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 -> Peer bd97ad2506a34450b6cf70414423528c (127.6.46.254:39189): Couldn't send request to peer bd97ad2506a34450b6cf70414423528c. Status: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 14:32:14.714486 18821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3092a8b44094e42b160704fc7b67d66. Latest consensus state: current_term: 2 leader_uuid: "f3092a8b44094e42b160704fc7b67d66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } } }
I20260810 14:32:14.714738 18821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:32:14.716208 18820 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f3092a8b44094e42b160704fc7b67d66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } } }
I20260810 14:32:14.716497 18820 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:14.717133 18821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f3092a8b44094e42b160704fc7b67d66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "bd97ad2506a34450b6cf70414423528c" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39189 } } peers { permanent_uuid: "5374e2ca086d4a858725cd7980b705d4" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 35217 } attrs { promote: false } } peers { permanent_uuid: "f3092a8b44094e42b160704fc7b67d66" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 38073 } attrs { promote: false } } }
I20260810 14:32:14.717363 18845 mvcc.cc:205] Tried to move back new op lower bound from 7316981082970963968 to 7316981082709073920. Current Snapshot: MvccSnapshot[applied={T|T < 7316981082970963968}]
I20260810 14:32:14.717417 18821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5374e2ca086d4a858725cd7980b705d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:32:14.732726 18824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.723928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:36107 (local address 127.0.0.1:43690)
0810 14:32:14.724436 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.724455 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:14.724559 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:14.726340 (+ 1781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:14.726345 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:14.726380 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:14.726711 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.726721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.728311 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.728317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.730391 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.730407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.732060 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.732066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.732229 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.732253 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:32:14.732341 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 14:32:14.732471 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394}
I20260810 14:32:14.732944 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.724301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.6.46.193:36107)
0810 14:32:14.725338 (+ 1037us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:14.725346 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:14.725366 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:32:14.725443 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:14.725449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:14.725528 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:14.726846 (+ 1318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:14.727026 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.728125 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.730621 (+ 2496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.731873 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.731990 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.732089 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:14.732502 (+ 413us) server_negotiation.cc:299] Negotiation successful
0810 14:32:14.732644 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":911,"thread_start_us":780,"threads_started":1}
I20260810 14:32:14.733624 18133 tablet_service.cc:1467] Processing CreateTablet for tablet 7d6839a6a8dd4622b130039271aa2649 (DEFAULT_TABLE table=test-table [id=06905c130573434c94aba60923ac972a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:14.734100 18133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d6839a6a8dd4622b130039271aa2649. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:14.737210 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.724707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.6.46.194:38723)
0810 14:32:14.731344 (+ 6637us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:14.731349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:14.731364 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:32:14.731418 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:14.731421 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:14.731475 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:14.731564 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:14.732139 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.733157 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.735449 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.736275 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.736438 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.736534 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:14.736787 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 14:32:14.736927 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6542,"thread_start_us":6451,"threads_started":1}
I20260810 14:32:14.737217 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.724300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:38723 (local address 127.0.0.1:33588)
0810 14:32:14.726063 (+ 1763us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.726081 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:14.726192 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:14.731597 (+ 5405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:14.731602 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:14.731626 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:14.731935 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.731947 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.733316 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.733321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.735288 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.735319 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.736477 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.736484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.736637 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.736669 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 14:32:14.736747 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:32:14.736871 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1609,"mutex_wait_us":72,"thread_start_us":1466,"threads_started":1}
I20260810 14:32:14.737896 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.723568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:44219 (local address 127.0.0.1:41548)
0810 14:32:14.723754 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.723771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:14.723871 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:14.732332 (+ 8461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:14.732336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:14.732362 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:14.732514 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.732525 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.733826 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.733831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.735721 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.735735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.737233 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.737239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.737420 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.737451 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 14:32:14.737529 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:32:14.737660 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":22}
I20260810 14:32:14.737943 18383 tablet_service.cc:1467] Processing CreateTablet for tablet 7d6839a6a8dd4622b130039271aa2649 (DEFAULT_TABLE table=test-table [id=06905c130573434c94aba60923ac972a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:14.738420 18383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d6839a6a8dd4622b130039271aa2649. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:14.738790 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.723713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41548 (local address 127.6.46.195:44219)
0810 14:32:14.732007 (+ 8294us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:14.732012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:14.732030 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:32:14.732099 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:14.732103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:14.732204 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:14.732305 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:14.732687 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.733667 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.735911 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.737057 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.738222 (+ 1165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.738323 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:14.738373 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 14:32:14.738503 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8178,"thread_start_us":8081,"threads_started":1}
I20260810 14:32:14.739452 18597 tablet_service.cc:1467] Processing CreateTablet for tablet 7d6839a6a8dd4622b130039271aa2649 (DEFAULT_TABLE table=test-table [id=06905c130573434c94aba60923ac972a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:14.739863 18597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d6839a6a8dd4622b130039271aa2649. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:14.740168 18858 tablet_bootstrap.cc:492] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c: Bootstrap starting.
I20260810 14:32:14.741649 18858 tablet_bootstrap.cc:654] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:14.743526 18858 tablet_bootstrap.cc:492] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c: No bootstrap required, opened a new log
I20260810 14:32:14.743673 18858 ts_tablet_manager.cc:1397] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260810 14:32:14.744270 18858 raft_consensus.cc:348] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } }
I20260810 14:32:14.744527 18858 raft_consensus.cc:374] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:14.744630 18858 raft_consensus.cc:729] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e4c148a686452caad3c9f8cb9bf56c, State: Initialized, Role: FOLLOWER
I20260810 14:32:14.744786 18860 tablet_bootstrap.cc:492] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d: Bootstrap starting.
I20260810 14:32:14.744891 18858 consensus_queue.cc:260] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } }
I20260810 14:32:14.746484 18860 tablet_bootstrap.cc:654] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:14.750488 18858 ts_tablet_manager.cc:1428] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260810 14:32:14.750735 18255 heartbeater.cc:503] Master 127.6.46.252:38073 was elected leader, sending a full tablet report...
I20260810 14:32:14.758764 18868 tablet_bootstrap.cc:492] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378: Bootstrap starting.
I20260810 14:32:14.761093 18868 tablet_bootstrap.cc:654] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:14.767012 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.745980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.193:54267)
0810 14:32:14.766672 (+ 20692us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.766776 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20546,"thread_start_us":20422,"threads_started":1}
I20260810 14:32:14.767970 18860 tablet_bootstrap.cc:492] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d: No bootstrap required, opened a new log
I20260810 14:32:14.768162 18860 ts_tablet_manager.cc:1397] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.015s
I20260810 14:32:14.769114 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.767731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.193:42143)
0810 14:32:14.768885 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.768978 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 14:32:14.769522 18860 raft_consensus.cc:348] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } }
I20260810 14:32:14.769767 18860 raft_consensus.cc:374] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:14.769869 18860 raft_consensus.cc:729] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7358de53372848ca93f3d28c6e674f2d, State: Initialized, Role: FOLLOWER
I20260810 14:32:14.769961 18868 tablet_bootstrap.cc:492] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378: No bootstrap required, opened a new log
I20260810 14:32:14.770138 18868 ts_tablet_manager.cc:1397] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260810 14:32:14.770190 18860 consensus_queue.cc:260] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } }
I20260810 14:32:14.770972 18868 raft_consensus.cc:348] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } }
I20260810 14:32:14.771222 18868 raft_consensus.cc:374] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:14.771323 18868 raft_consensus.cc:729] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031348ed4bcd474687b9a337f7cb4378, State: Initialized, Role: FOLLOWER
I20260810 14:32:14.771564 18868 consensus_queue.cc:260] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } }
I20260810 14:32:14.775020 18868 ts_tablet_manager.cc:1428] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260810 14:32:14.775286 18702 heartbeater.cc:503] Master 127.6.46.252:38073 was elected leader, sending a full tablet report...
I20260810 14:32:14.775416 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.772660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.195:44771)
0810 14:32:14.774964 (+ 2304us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.775114 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2164,"thread_start_us":2023,"threads_started":1}
I20260810 14:32:14.776099 18860 ts_tablet_manager.cc:1428] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d: Time spent starting tablet: real 0.008s user 0.001s sys 0.006s
I20260810 14:32:14.778642 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.776015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.194:57589)
0810 14:32:14.778209 (+ 2194us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.778376 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2033,"threads_started":1}
I20260810 14:32:14.780370 18488 heartbeater.cc:503] Master 127.6.46.252:38073 was elected leader, sending a full tablet report...
I20260810 14:32:14.781025 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.780589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.194:50007)
0810 14:32:14.780818 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.780901 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260810 14:32:14.782150 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.780622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.195:43503)
0810 14:32:14.781886 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.781972 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260810 14:32:14.796955 18872 raft_consensus.cc:482] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:32:14.797138 18872 raft_consensus.cc:504] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } }
I20260810 14:32:14.800293 18872 leader_election.cc:288] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031348ed4bcd474687b9a337f7cb4378 (127.6.46.195:44219), a6e4c148a686452caad3c9f8cb9bf56c (127.6.46.193:36107)
I20260810 14:32:14.805764 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.799133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:55889 (local address 127.6.46.195:44219)
0810 14:32:14.799321 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:14.799326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:14.799347 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:32:14.799458 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:14.799463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:14.799536 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:14.799625 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:14.800392 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.801340 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.803553 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.804557 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.804738 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.804819 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:14.805347 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 14:32:14.805489 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:32:14.806434 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.799000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:44219 (local address 127.6.46.194:55889)
0810 14:32:14.799178 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.799199 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:14.799335 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:14.799788 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:14.799792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:14.799854 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:14.800202 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.800221 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.801513 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.801518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.803409 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.803422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.804943 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.804949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.805144 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.805180 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 14:32:14.805951 (+ 771us) client_negotiation.cc:240] Negotiation successful
0810 14:32:14.806139 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 14:32:14.806972 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.800154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:46153 (local address 127.6.46.193:36107)
0810 14:32:14.800351 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:14.800356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:14.800426 (+ 70us) server_negotiation.cc:407] Connection header received
0810 14:32:14.800678 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:14.800684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:14.800758 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:14.800838 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:14.801495 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.802518 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.804573 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.805728 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.806103 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.806185 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:14.806398 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 14:32:14.806740 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:32:14.807098 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.799655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:36107 (local address 127.6.46.194:46153)
0810 14:32:14.800344 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.800361 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:14.800532 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:14.801004 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:14.801009 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:14.801035 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:14.801345 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.801358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.802671 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.802676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.804431 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:14.804444 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.806062 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.806069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:14.806255 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.806284 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 14:32:14.806363 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:32:14.806483 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":364,"threads_started":1}
I20260810 14:32:14.807085 18630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d6839a6a8dd4622b130039271aa2649" candidate_uuid: "7358de53372848ca93f3d28c6e674f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031348ed4bcd474687b9a337f7cb4378" is_pre_election: true
I20260810 14:32:14.807343 18630 raft_consensus.cc:2454] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7358de53372848ca93f3d28c6e674f2d in term 0.
I20260810 14:32:14.808054 18307 leader_election.cc:302] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031348ed4bcd474687b9a337f7cb4378, 7358de53372848ca93f3d28c6e674f2d; no voters:
I20260810 14:32:14.808312 18174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d6839a6a8dd4622b130039271aa2649" candidate_uuid: "7358de53372848ca93f3d28c6e674f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" is_pre_election: true
I20260810 14:32:14.808526 18872 raft_consensus.cc:2781] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:32:14.808712 18872 raft_consensus.cc:482] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:32:14.808719 18174 raft_consensus.cc:2454] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7358de53372848ca93f3d28c6e674f2d in term 0.
I20260810 14:32:14.808811 18872 raft_consensus.cc:3037] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:14.812105 18872 raft_consensus.cc:504] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } }
I20260810 14:32:14.812971 18872 leader_election.cc:288] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [CANDIDATE]: Term 1 election: Requested vote from peers 031348ed4bcd474687b9a337f7cb4378 (127.6.46.195:44219), a6e4c148a686452caad3c9f8cb9bf56c (127.6.46.193:36107)
I20260810 14:32:14.813141 18630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d6839a6a8dd4622b130039271aa2649" candidate_uuid: "7358de53372848ca93f3d28c6e674f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031348ed4bcd474687b9a337f7cb4378"
I20260810 14:32:14.813287 18174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d6839a6a8dd4622b130039271aa2649" candidate_uuid: "7358de53372848ca93f3d28c6e674f2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e4c148a686452caad3c9f8cb9bf56c"
I20260810 14:32:14.813354 18630 raft_consensus.cc:3037] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:14.813449 18174 raft_consensus.cc:3037] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:14.816277 18630 raft_consensus.cc:2454] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7358de53372848ca93f3d28c6e674f2d in term 1.
I20260810 14:32:14.816308 18174 raft_consensus.cc:2454] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7358de53372848ca93f3d28c6e674f2d in term 1.
I20260810 14:32:14.816670 18304 leader_election.cc:302] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7358de53372848ca93f3d28c6e674f2d, a6e4c148a686452caad3c9f8cb9bf56c; no voters:
I20260810 14:32:14.816962 18872 raft_consensus.cc:2781] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:14.817423 18872 raft_consensus.cc:686] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [term 1 LEADER]: Becoming Leader. State: Replica: 7358de53372848ca93f3d28c6e674f2d, State: Running, Role: LEADER
I20260810 14:32:14.817764 18872 consensus_queue.cc:237] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } }
I20260810 14:32:14.818740 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.818256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.194:50465)
0810 14:32:14.818452 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.818572 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 14:32:14.821400 17950 catalog_manager.cc:5495] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7358de53372848ca93f3d28c6e674f2d (127.6.46.194). New cstate: current_term: 1 leader_uuid: "7358de53372848ca93f3d28c6e674f2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7358de53372848ca93f3d28c6e674f2d" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38723 } health_report { overall_health: HEALTHY } } }
I20260810 14:32:14.838315 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.833710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.6.46.194:38723)
0810 14:32:14.833889 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:14.833894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:14.833910 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:32:14.834094 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:14.834099 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:14.834182 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:14.834296 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:32:14.834579 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.835272 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.836509 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.836909 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.836996 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.837325 (+ 329us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:32:14.837824 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:32:14.837932 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:14.837968 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 14:32:14.838058 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:32:14.841881 18630 raft_consensus.cc:1261] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378 [term 1 FOLLOWER]: Refusing update from remote peer 7358de53372848ca93f3d28c6e674f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:14.842391 18174 raft_consensus.cc:1261] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c [term 1 FOLLOWER]: Refusing update from remote peer 7358de53372848ca93f3d28c6e674f2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:14.842630 18880 consensus_queue.cc:1035] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "031348ed4bcd474687b9a337f7cb4378" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:14.843096 18872 consensus_queue.cc:1035] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e4c148a686452caad3c9f8cb9bf56c" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 36107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:14.844877 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.844443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.195:43989)
0810 14:32:14.844596 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.844710 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 14:32:14.846122 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.845734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.193:42637)
0810 14:32:14.845860 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.845952 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 14:32:14.853430 18885 mvcc.cc:205] Tried to move back new op lower bound from 7316981083506528256 to 7316981083417726976. Current Snapshot: MvccSnapshot[applied={T|T < 7316981083506528256}]
I20260810 14:32:14.854952 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.854501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.193:44637)
0810 14:32:14.854742 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.854823 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260810 14:32:14.855023 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.854573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.194:56111)
0810 14:32:14.854788 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.854865 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260810 14:32:14.856072 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.855671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.194:57723)
0810 14:32:14.855864 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.855941 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 14:32:14.858952 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.858504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.6.46.195:37593)
0810 14:32:14.858715 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:14.858800 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260810 14:32:14.868022 17947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:14.868258 17947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:14.873881 17947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3092a8b44094e42b160704fc7b67d66: Sending DeleteTablet for 3 replicas of tablet 7d6839a6a8dd4622b130039271aa2649
I20260810 14:32:14.874996 18133 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d6839a6a8dd4622b130039271aa2649 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:14 UTC) from {username='oryx', principal='oryx/127.6.46.252@KRBTEST.COM'} at 127.0.0.1:43690
I20260810 14:32:14.875169 18597 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d6839a6a8dd4622b130039271aa2649 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:14 UTC) from {username='oryx', principal='oryx/127.6.46.252@KRBTEST.COM'} at 127.0.0.1:41548
I20260810 14:32:14.875561 18383 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d6839a6a8dd4622b130039271aa2649 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:14 UTC) from {username='oryx', principal='oryx/127.6.46.252@KRBTEST.COM'} at 127.0.0.1:33588
I20260810 14:32:14.875653 18896 tablet_replica.cc:330] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c: stopping tablet replica
I20260810 14:32:14.875823 18896 raft_consensus.cc:2229] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:14.876123 18896 raft_consensus.cc:2258] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:14.876331 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 17894
I20260810 14:32:14.879088 18897 tablet_replica.cc:330] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378: stopping tablet replica
I20260810 14:32:14.879285 18897 raft_consensus.cc:2229] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:14.879565 18897 raft_consensus.cc:2258] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:14.880183 18899 tablet_replica.cc:330] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d: stopping tablet replica
I20260810 14:32:14.880379 18899 raft_consensus.cc:2229] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:32:14.880970 18899 raft_consensus.cc:2258] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:14.882192 18899 ts_tablet_manager.cc:1905] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:14.882192 18897 ts_tablet_manager.cc:1905] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:14.883517 18896 ts_tablet_manager.cc:1905] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:14.888756 18897 ts_tablet_manager.cc:1918] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:14.888899 18897 log.cc:1196] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/7d6839a6a8dd4622b130039271aa2649
I20260810 14:32:14.889405 18897 ts_tablet_manager.cc:1939] T 7d6839a6a8dd4622b130039271aa2649 P 031348ed4bcd474687b9a337f7cb4378: Deleting consensus metadata
I20260810 14:32:14.889616 18899 ts_tablet_manager.cc:1918] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:14.889818 18899 log.cc:1196] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/7d6839a6a8dd4622b130039271aa2649
I20260810 14:32:14.890403 18899 ts_tablet_manager.cc:1939] T 7d6839a6a8dd4622b130039271aa2649 P 7358de53372848ca93f3d28c6e674f2d: Deleting consensus metadata
I20260810 14:32:14.890594 17930 catalog_manager.cc:4841] TS 031348ed4bcd474687b9a337f7cb4378 (127.6.46.195:44219): tablet 7d6839a6a8dd4622b130039271aa2649 (table test-table [id=06905c130573434c94aba60923ac972a]) successfully deleted
I20260810 14:32:14.891604 17930 catalog_manager.cc:4841] TS 7358de53372848ca93f3d28c6e674f2d (127.6.46.194:38723): tablet 7d6839a6a8dd4622b130039271aa2649 (table test-table [id=06905c130573434c94aba60923ac972a]) successfully deleted
I20260810 14:32:14.892473 18896 ts_tablet_manager.cc:1918] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:14.892591 18896 log.cc:1196] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/7d6839a6a8dd4622b130039271aa2649
I20260810 14:32:14.892938 18896 ts_tablet_manager.cc:1939] T 7d6839a6a8dd4622b130039271aa2649 P a6e4c148a686452caad3c9f8cb9bf56c: Deleting consensus metadata
I20260810 14:32:14.893811 17922 catalog_manager.cc:4841] TS a6e4c148a686452caad3c9f8cb9bf56c (127.6.46.193:36107): tablet 7d6839a6a8dd4622b130039271aa2649 (table test-table [id=06905c130573434c94aba60923ac972a]) successfully deleted
I20260810 14:32:14.908412 18855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:14.877238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.6.46.193:36107)
0810 14:32:14.877409 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:14.877415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:14.894507 (+ 17092us) server_negotiation.cc:407] Connection header received
0810 14:32:14.894697 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:14.894703 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:14.894799 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:14.894915 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:14.896273 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.897377 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.898557 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:14.899240 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:14.899365 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:14.901703 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:14.901722 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:14.901743 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:14.901783 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:14.905376 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:14.905970 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:14.905980 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:14.905984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:14.906207 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:14.906664 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:14.906669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:14.906673 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:14.907205 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:14.907537 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:14.907762 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 14:32:14.907980 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:32:14.909102 18247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:15.027845 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 17894
I20260810 14:32:15.071283 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 18275
I20260810 14:32:15.094367 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:15.072058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.6.46.194:38723)
0810 14:32:15.072222 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:15.072226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:15.072244 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:32:15.072415 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:15.072419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:15.072480 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:15.072562 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:15.074306 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:15.075165 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:15.076928 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:15.077369 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:15.077470 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:15.087707 (+ 10237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:15.087721 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:15.087733 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:15.087772 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:15.091447 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:15.092117 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:15.092126 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:15.092130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:15.092325 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:15.092670 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:15.092675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:15.092678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:15.093105 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:15.093454 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:15.093684 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 14:32:15.093942 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:32:15.095150 18482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:15.111616 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:15.111207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:42436)
0810 14:32:15.111383 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:15.111461 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 14:32:15.112495 18822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:15.112165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:39189 (local address 127.0.0.1:42444)
0810 14:32:15.112307 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:15.112390 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 14:32:15.209501 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 18275
I20260810 14:32:15.232173 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 18506
I20260810 14:32:15.244755 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:15.232961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41554 (local address 127.6.46.195:44219)
0810 14:32:15.233138 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:15.233142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:15.233161 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:32:15.233300 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:15.233304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:15.233366 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:15.233445 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:15.234787 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:15.235528 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:15.236702 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:15.237093 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:15.237185 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:15.239656 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:15.239671 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:15.239686 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:15.239730 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:15.242401 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:15.243009 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:15.243017 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:15.243019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:15.243158 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:15.243427 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:15.243430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:15.243432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:15.243760 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:15.243972 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:15.244259 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 14:32:15.244449 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 14:32:15.245409 18691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:15.335949 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 18506
I20260810 14:32:15.360059 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 17372
I20260810 14:32:15.361137 17666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:15.476372 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 17372
I20260810 14:32:15.501006 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 17552
I20260810 14:32:15.501958 17996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:15.621121 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 17552
2026-08-10T14:32:15Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9479 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:32:15 dist-test-slave-q9h9 krb5kdc[18917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:32:15 dist-test-slave-q9h9 krb5kdc[18917](info): set up 2 sockets
Aug 10 14:32:15 dist-test-slave-q9h9 krb5kdc[18917](info): commencing operation
krb5kdc: starting...
W20260810 14:32:16.857481 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.182s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:32:17 dist-test-slave-q9h9 krb5kdc[18917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:32:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:32:17Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.254.keytab.
Entry for principal oryx/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.254.keytab.
I20260810 14:32:17.077268 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36175
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.254.keytab
--principal=oryx/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:36175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:17.390233 18944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:17.390513 18944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:17.390651 18944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:17.397660 18944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:32:17.397760 18944 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 14:32:17.397830 18944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:17.397912 18944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:17.397974 18944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:32:17.398088 18944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:32:17.408124 18944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:36175
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.254.keytab
--principal=oryx/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:17.409299 18944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:17.410617 18944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:17.421703 18957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:17.421671 18956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:17.422235 18960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:17.422406 18944 server_base.cc:1034] running on GCE node
I20260810 14:32:17.423143 18944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:17.424410 18944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:17.425623 18944 hybrid_clock.cc:648] HybridClock initialized: now 1786372337425553 us; error 42 us; skew 500 ppm
I20260810 14:32:17.425931 18944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:17 dist-test-slave-q9h9 krb5kdc[18917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372337, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:17.428907 18944 init.cc:377] Logged in from keytab as oryx/127.6.46.254@KRBTEST.COM (short username oryx)
I20260810 14:32:17.430291 18944 webserver.cc:460] Webserver started at http://127.6.46.254:36241/ using document root <none> and password file <none>
I20260810 14:32:17.430944 18944 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:17.431044 18944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:17.431350 18944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:17.434194 18944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "1e51fef1490e4a1f9644678f1d564ea1"
format_stamp: "Formatted at 2026-08-10 14:32:17 on dist-test-slave-q9h9"
server_key: "c0eaf4f977bac6db911d320da58461fd"
server_key_iv: "11551b7f9f7cbd63d5899c01f7b5b4ea"
server_key_version: "encryptionkey@0"
I20260810 14:32:17.434979 18944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "1e51fef1490e4a1f9644678f1d564ea1"
format_stamp: "Formatted at 2026-08-10 14:32:17 on dist-test-slave-q9h9"
server_key: "c0eaf4f977bac6db911d320da58461fd"
server_key_iv: "11551b7f9f7cbd63d5899c01f7b5b4ea"
server_key_version: "encryptionkey@0"
I20260810 14:32:17.439741 18944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:32:17.442971 18971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:17.444203 18944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:32:17.444367 18944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "1e51fef1490e4a1f9644678f1d564ea1"
format_stamp: "Formatted at 2026-08-10 14:32:17 on dist-test-slave-q9h9"
server_key: "c0eaf4f977bac6db911d320da58461fd"
server_key_iv: "11551b7f9f7cbd63d5899c01f7b5b4ea"
server_key_version: "encryptionkey@0"
I20260810 14:32:17.444566 18944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:17.457573 18944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:17.461354 18944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:17.461601 18944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:17.481194 18944 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:36175
I20260810 14:32:17.481200 19069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:36175 every 8 connection(s)
I20260810 14:32:17.482484 18944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:32:17.485924 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 18944
I20260810 14:32:17.486208 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:32:17.486253 19070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:17.487016 6331 external_mini_cluster.cc:1428] Setting key eac0ded35d90ecf1bb3718278fae4bd7
I20260810 14:32:17.498421 19070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1: Bootstrap starting.
Aug 10 14:32:17 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
I20260810 14:32:17.501809 19070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:17.502972 19070 log.cc:824] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:17.505947 19070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1: No bootstrap required, opened a new log
I20260810 14:32:17.508260 19073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:17.489058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55356 (local address 127.6.46.254:36175)
0810 14:32:17.489843 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:17.489854 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:17.489892 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:32:17.491440 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:17.491464 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:17.491816 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:17.492178 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:17.493604 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:17.494886 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:17.496092 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:17.496756 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:17.496911 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:17.501065 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:17.501072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:17.501085 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:17.501119 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:17.504375 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:17.505136 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:17.505141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:17.505146 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:17.505310 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:17.505709 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:17.505712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:17.505714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:17.506248 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:17.506453 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:17.506909 (+ 456us) server_negotiation.cc:299] Negotiation successful
0810 14:32:17.507220 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":357,"threads_started":1}
I20260810 14:32:17.513486 19070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e51fef1490e4a1f9644678f1d564ea1" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36175 } }
I20260810 14:32:17.513806 19070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:17.513902 19070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e51fef1490e4a1f9644678f1d564ea1, State: Initialized, Role: FOLLOWER
I20260810 14:32:17.514443 19070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e51fef1490e4a1f9644678f1d564ea1" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36175 } }
I20260810 14:32:17.514667 19070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:32:17.514807 19070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:32:17.514956 19070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:17.516897 19070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e51fef1490e4a1f9644678f1d564ea1" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36175 } }
I20260810 14:32:17.517349 19070 leader_election.cc:302] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e51fef1490e4a1f9644678f1d564ea1; no voters:
I20260810 14:32:17.517851 19070 leader_election.cc:288] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:32:17.517948 19076 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:17.519338 19076 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [term 1 LEADER]: Becoming Leader. State: Replica: 1e51fef1490e4a1f9644678f1d564ea1, State: Running, Role: LEADER
I20260810 14:32:17.519805 19076 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e51fef1490e4a1f9644678f1d564ea1" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36175 } }
I20260810 14:32:17.519938 19070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:32:17.522495 19077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e51fef1490e4a1f9644678f1d564ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e51fef1490e4a1f9644678f1d564ea1" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36175 } } }
I20260810 14:32:17.522814 19077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:17.522306 19081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e51fef1490e4a1f9644678f1d564ea1. Latest consensus state: current_term: 1 leader_uuid: "1e51fef1490e4a1f9644678f1d564ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e51fef1490e4a1f9644678f1d564ea1" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36175 } } }
I20260810 14:32:17.523088 19081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:17.523733 19090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:32:17.529269 19090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:32:17.537325 19090 catalog_manager.cc:1347] Generated new cluster ID: 41f8d67e899d4917a75bc101cba75bd6
I20260810 14:32:17.537425 19090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:32:17.552490 19090 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:32:17.553653 19090 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:32:17.567319 19090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1: Generated new TSK 0
I20260810 14:32:17.568024 19090 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab.
Entry for principal oryx/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab.
I20260810 14:32:17.623442 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36175
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab
--principal=oryx/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:17.954949 19105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:17.955214 19105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:17.955348 19105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:17.962285 19105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:17.962389 19105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:17.962508 19105 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:32:17.962623 19105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:32:17.971824 19105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab
--principal=oryx/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.46.254:36175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:17.972966 19105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:17.974246 19105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:17.986383 19114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:17.986490 19113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:17.986342 19116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:17.987051 19105 server_base.cc:1034] running on GCE node
I20260810 14:32:17.987577 19105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:17.988591 19105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:17.989836 19105 hybrid_clock.cc:648] HybridClock initialized: now 1786372337989762 us; error 50 us; skew 500 ppm
I20260810 14:32:17.990197 19105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:17 dist-test-slave-q9h9 krb5kdc[18917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372337, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:17.993101 19105 init.cc:377] Logged in from keytab as oryx/127.6.46.193@KRBTEST.COM (short username oryx)
I20260810 14:32:17.994427 19105 webserver.cc:460] Webserver started at http://127.6.46.193:32851/ using document root <none> and password file <none>
I20260810 14:32:17.995085 19105 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:17.995179 19105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:17.995487 19105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:17.998307 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "68ab8c71129b4a3aaacbadf8447307a2"
format_stamp: "Formatted at 2026-08-10 14:32:17 on dist-test-slave-q9h9"
server_key: "194f022531dd3d9e7e4a97c470eca970"
server_key_iv: "1a41b20a76c986b9ea7d01290071891e"
server_key_version: "encryptionkey@0"
I20260810 14:32:17.999059 19105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "68ab8c71129b4a3aaacbadf8447307a2"
format_stamp: "Formatted at 2026-08-10 14:32:17 on dist-test-slave-q9h9"
server_key: "194f022531dd3d9e7e4a97c470eca970"
server_key_iv: "1a41b20a76c986b9ea7d01290071891e"
server_key_version: "encryptionkey@0"
I20260810 14:32:18.003831 19105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:32:18.007056 19126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:18.008370 19105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:32:18.008504 19105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "68ab8c71129b4a3aaacbadf8447307a2"
format_stamp: "Formatted at 2026-08-10 14:32:17 on dist-test-slave-q9h9"
server_key: "194f022531dd3d9e7e4a97c470eca970"
server_key_iv: "1a41b20a76c986b9ea7d01290071891e"
server_key_version: "encryptionkey@0"
I20260810 14:32:18.008720 19105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:18.024348 19105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:18.027963 19105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:18.028198 19105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:18.030761 19105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:18.030859 19105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:18.031011 19105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:18.031092 19105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:32:18 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372337, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
I20260810 14:32:18.052947 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:18.031223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:51477 (local address 127.6.46.254:36175)
0810 14:32:18.031930 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:18.031934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:18.032799 (+ 865us) server_negotiation.cc:407] Connection header received
0810 14:32:18.034789 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:18.034794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:18.034885 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:18.034980 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:18.036911 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.037844 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.040241 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.040819 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.040932 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:18.045879 (+ 4947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:18.045886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:18.045891 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:18.045934 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:18.049096 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:18.050049 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:18.050054 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:18.050058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:18.050189 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:18.050757 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:18.050761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:18.050763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:18.051162 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:18.051305 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:18.052264 (+ 959us) server_negotiation.cc:299] Negotiation successful
0810 14:32:18.052519 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":537,"threads_started":1}
I20260810 14:32:18.053846 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:18.031847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36175 (local address 127.6.46.193:51477)
0810 14:32:18.032659 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:18.032694 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:18.034508 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:18.035209 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:18.035216 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:18.035888 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:18.036680 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:18.036706 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.037995 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.038004 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:18.040088 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:18.040118 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.041069 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.041078 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:18.041405 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:18.042277 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:18.042316 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:18.045631 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:18.049285 (+ 3654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:18.049294 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:18.049310 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:18.049905 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:18.050302 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:18.050307 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:18.050311 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:18.050598 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:18.051276 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:18.051284 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:18.051805 (+ 521us) client_negotiation.cc:769] Sending connection context
0810 14:32:18.052117 (+ 312us) client_negotiation.cc:240] Negotiation successful
0810 14:32:18.052686 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":357,"threads_started":1}
I20260810 14:32:18.072907 19105 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:35025
I20260810 14:32:18.072929 19317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:35025 every 8 connection(s)
I20260810 14:32:18.073374 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:18.059404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:58477 (local address 127.6.46.254:36175)
0810 14:32:18.059558 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:18.059564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:18.059997 (+ 433us) server_negotiation.cc:407] Connection header received
0810 14:32:18.060164 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:18.060169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:18.060252 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:18.060350 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:18.061772 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.062725 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.064786 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.065195 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.065309 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:18.067785 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:18.067791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:18.067794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:18.067827 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:18.070749 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:18.071456 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:18.071461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:18.071465 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:18.071593 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:18.071945 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:18.071950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:18.071953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:18.072291 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:18.072421 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:18.072839 (+ 418us) server_negotiation.cc:299] Negotiation successful
0810 14:32:18.073040 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:32:18.073591 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:18.059297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36175 (local address 127.6.46.193:58477)
0810 14:32:18.059902 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:18.059916 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:18.060048 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:18.060412 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:18.060417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:18.061139 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:18.061610 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:18.061624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.062869 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.062874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:18.064641 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:18.064659 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.065398 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.065405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:18.065558 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:18.066278 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:18.066301 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:18.067614 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:18.070873 (+ 3259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:18.070879 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:18.070883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:18.071351 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:18.071705 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:18.071710 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:18.071714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:18.071846 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:18.072401 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:18.072405 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:18.072589 (+ 184us) client_negotiation.cc:769] Sending connection context
0810 14:32:18.072739 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 14:32:18.073128 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":351,"threads_started":1}
I20260810 14:32:18.074338 19105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:32:18.076234 19319 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36175
I20260810 14:32:18.076452 19319 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:18.077090 19319 heartbeater.cc:511] Master 127.6.46.254:36175 requested a full tablet report, sending...
I20260810 14:32:18.078943 18995 ts_manager.cc:194] Registered new tserver with Master: 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193:35025)
I20260810 14:32:18.080294 18995 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:51477
I20260810 14:32:18.081584 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 19105
I20260810 14:32:18.081779 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:32:18.082445 6331 external_mini_cluster.cc:1428] Setting key 3365280f1bf717b45460bdee5ac6835a
WARNING: no policy specified for oryx/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab.
Entry for principal oryx/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab.
I20260810 14:32:18.130539 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36175
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab
--principal=oryx/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:18.445004 19326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:18.445264 19326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:18.445366 19326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:18.452368 19326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:18.452466 19326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:18.452550 19326 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:32:18.452626 19326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:32:18.461910 19326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab
--principal=oryx/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.46.254:36175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:18.463055 19326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:18.464327 19326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:18.476302 19338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:18.476388 19332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:18.476348 19333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:18.477118 19326 server_base.cc:1034] running on GCE node
I20260810 14:32:18.477687 19326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:18.478552 19326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:18.479769 19326 hybrid_clock.cc:648] HybridClock initialized: now 1786372338479705 us; error 32 us; skew 500 ppm
I20260810 14:32:18.480046 19326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:18 dist-test-slave-q9h9 krb5kdc[18917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372338, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:18.483304 19326 init.cc:377] Logged in from keytab as oryx/127.6.46.194@KRBTEST.COM (short username oryx)
I20260810 14:32:18.484903 19326 webserver.cc:460] Webserver started at http://127.6.46.194:39437/ using document root <none> and password file <none>
I20260810 14:32:18.485522 19326 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:18.485622 19326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:18.485920 19326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:18.488657 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "d92948900f73475b81fd739707311954"
format_stamp: "Formatted at 2026-08-10 14:32:18 on dist-test-slave-q9h9"
server_key: "a84dd104de970c790edf31f405de099d"
server_key_iv: "d5e48600aa022a2cc46d9ae752337d4c"
server_key_version: "encryptionkey@0"
I20260810 14:32:18.489439 19326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "d92948900f73475b81fd739707311954"
format_stamp: "Formatted at 2026-08-10 14:32:18 on dist-test-slave-q9h9"
server_key: "a84dd104de970c790edf31f405de099d"
server_key_iv: "d5e48600aa022a2cc46d9ae752337d4c"
server_key_version: "encryptionkey@0"
I20260810 14:32:18.494311 19326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20260810 14:32:18.497506 19352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:18.498730 19326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:32:18.498870 19326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "d92948900f73475b81fd739707311954"
format_stamp: "Formatted at 2026-08-10 14:32:18 on dist-test-slave-q9h9"
server_key: "a84dd104de970c790edf31f405de099d"
server_key_iv: "d5e48600aa022a2cc46d9ae752337d4c"
server_key_version: "encryptionkey@0"
I20260810 14:32:18.499056 19326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:18.512213 19326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:18.516126 19326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:18.516360 19326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:18.518908 19326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:18.519009 19326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:18.519178 19326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:18.519253 19326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:32:18 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372338, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
I20260810 14:32:18.540373 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:18.519230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:53245 (local address 127.6.46.254:36175)
0810 14:32:18.519399 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:18.519404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:18.520846 (+ 1442us) server_negotiation.cc:407] Connection header received
0810 14:32:18.522958 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:18.522965 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:18.523068 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:18.523177 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:18.525110 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.526273 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.528608 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.529075 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.529189 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:18.533656 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:18.533663 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:18.533667 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:18.533700 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:18.536767 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:18.537848 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:18.537854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:18.537857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:18.537986 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:18.538528 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:18.538533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:18.538536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:18.538868 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:18.539013 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:18.539966 (+ 953us) server_negotiation.cc:299] Negotiation successful
0810 14:32:18.540136 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:32:18.541850 19367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:18.519842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36175 (local address 127.6.46.194:53245)
0810 14:32:18.520681 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:18.520750 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:18.522633 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:18.523376 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:18.523383 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:18.524100 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:18.524879 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:18.524899 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.526448 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.526453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:18.528403 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:18.528421 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.529320 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.529328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:18.529564 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:18.530444 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:18.530478 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:18.533423 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:18.536941 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:18.536950 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:18.536970 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:18.537721 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:18.538124 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:18.538129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:18.538133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:18.538360 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:18.539022 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:18.539040 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:18.539500 (+ 460us) client_negotiation.cc:769] Sending connection context
0810 14:32:18.539841 (+ 341us) client_negotiation.cc:240] Negotiation successful
0810 14:32:18.540413 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":374,"threads_started":1}
I20260810 14:32:18.560079 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:18.546827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:42957 (local address 127.6.46.254:36175)
0810 14:32:18.546981 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:18.546987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:18.547877 (+ 890us) server_negotiation.cc:407] Connection header received
0810 14:32:18.548017 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:18.548023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:18.548083 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:18.548161 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:18.549054 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.549749 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.551449 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.552017 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.552138 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:18.554404 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:18.554409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:18.554412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:18.554441 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:18.557299 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:18.557957 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:18.557962 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:18.557966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:18.558164 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:18.558593 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:18.558599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:18.558603 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:18.559018 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:18.559163 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:18.559562 (+ 399us) server_negotiation.cc:299] Negotiation successful
0810 14:32:18.559770 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:32:18.560302 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:18.547011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36175 (local address 127.6.46.194:42957)
0810 14:32:18.547767 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:18.547781 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:18.547896 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:18.548204 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:18.548209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:18.548647 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:18.548917 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:18.548925 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.549880 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.549886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:18.551252 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:18.551271 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:18.552389 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:18.552396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:18.552535 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:18.553171 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:18.553188 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:18.554179 (+ 991us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:18.557421 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:18.557426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:18.557431 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:18.557817 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:18.558314 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:18.558319 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:18.558322 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:18.558487 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:18.559137 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:18.559142 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:18.559332 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 14:32:18.559460 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 14:32:18.559819 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":498,"threads_started":1}
I20260810 14:32:18.568282 19326 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:37085
I20260810 14:32:18.568321 19549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:37085 every 8 connection(s)
I20260810 14:32:18.569360 19326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:32:18.571175 19552 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36175
I20260810 14:32:18.571374 19552 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:18.571954 19552 heartbeater.cc:511] Master 127.6.46.254:36175 requested a full tablet report, sending...
I20260810 14:32:18.573189 18995 ts_manager.cc:194] Registered new tserver with Master: d92948900f73475b81fd739707311954 (127.6.46.194:37085)
I20260810 14:32:18.573554 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 19326
I20260810 14:32:18.573760 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:32:18.574321 6331 external_mini_cluster.cc:1428] Setting key 8267fb2ef4bd265324f51bde2ff423b7
I20260810 14:32:18.574283 18995 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:53245
WARNING: no policy specified for oryx/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab.
Entry for principal oryx/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab.
I20260810 14:32:18.624660 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36175
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab
--principal=oryx/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:18.941926 19557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:18.942229 19557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:18.942371 19557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:18.950364 19557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:18.950466 19557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:18.950569 19557 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 14:32:18.950678 19557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:32:18.960309 19557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab
--principal=oryx/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.46.254:36175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:18.961505 19557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:18.962868 19557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:18.975214 19564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:18.975261 19569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:18.975287 19563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:18.976255 19557 server_base.cc:1034] running on GCE node
I20260810 14:32:18.976807 19557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:18.977613 19557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:18.978808 19557 hybrid_clock.cc:648] HybridClock initialized: now 1786372338978746 us; error 36 us; skew 500 ppm
I20260810 14:32:18.979084 19557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:18 dist-test-slave-q9h9 krb5kdc[18917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372338, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:18.982079 19557 init.cc:377] Logged in from keytab as oryx/127.6.46.195@KRBTEST.COM (short username oryx)
I20260810 14:32:18.983424 19557 webserver.cc:460] Webserver started at http://127.6.46.195:36035/ using document root <none> and password file <none>
I20260810 14:32:18.984063 19557 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:18.984160 19557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:18.984459 19557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:18.987294 19557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "6521c3bfbdbf4fc68aaad46b613e6020"
format_stamp: "Formatted at 2026-08-10 14:32:18 on dist-test-slave-q9h9"
server_key: "f8d34ffa8b0e6a84a4184424456ffcf0"
server_key_iv: "2071e7bd656a302a4b43f811433378da"
server_key_version: "encryptionkey@0"
I20260810 14:32:18.988034 19557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "6521c3bfbdbf4fc68aaad46b613e6020"
format_stamp: "Formatted at 2026-08-10 14:32:18 on dist-test-slave-q9h9"
server_key: "f8d34ffa8b0e6a84a4184424456ffcf0"
server_key_iv: "2071e7bd656a302a4b43f811433378da"
server_key_version: "encryptionkey@0"
I20260810 14:32:18.992795 19557 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:32:18.996076 19580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:18.997291 19557 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:32:18.997432 19557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "6521c3bfbdbf4fc68aaad46b613e6020"
format_stamp: "Formatted at 2026-08-10 14:32:18 on dist-test-slave-q9h9"
server_key: "f8d34ffa8b0e6a84a4184424456ffcf0"
server_key_iv: "2071e7bd656a302a4b43f811433378da"
server_key_version: "encryptionkey@0"
I20260810 14:32:18.997617 19557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:19.010212 19557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:19.013880 19557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:19.014138 19557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:19.016579 19557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:19.016673 19557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:19.016816 19557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:19.016888 19557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:32:19 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372338, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
I20260810 14:32:19.038282 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.017125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:35119 (local address 127.6.46.254:36175)
0810 14:32:19.017347 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:19.017353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:19.018626 (+ 1273us) server_negotiation.cc:407] Connection header received
0810 14:32:19.020786 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:19.020793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:19.020913 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:19.021034 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:19.023129 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.024063 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.026852 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.027362 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.027485 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.031521 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:19.031528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:19.031532 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:19.031574 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:19.034989 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:19.035861 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:19.035867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:19.035871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:19.036037 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:19.036586 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:19.036590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:19.036592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:19.036868 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:19.036979 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:19.037795 (+ 816us) server_negotiation.cc:299] Negotiation successful
0810 14:32:19.037963 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 14:32:19.039716 19594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.017663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36175 (local address 127.6.46.195:35119)
0810 14:32:19.018467 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:19.018513 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:19.020463 (+ 1950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:19.021320 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:19.021331 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:19.022096 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:19.022944 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.022970 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.024255 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.024260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.026708 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.026727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.027816 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.027825 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.028016 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.028706 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:19.028727 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:19.031250 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:19.035120 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:19.035132 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:19.035156 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:19.035737 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:19.036163 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:19.036168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:19.036172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:19.036461 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:19.036981 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:19.036989 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:19.037401 (+ 412us) client_negotiation.cc:769] Sending connection context
0810 14:32:19.037705 (+ 304us) client_negotiation.cc:240] Negotiation successful
0810 14:32:19.038268 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":379,"threads_started":1}
I20260810 14:32:19.062901 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.048798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:60215 (local address 127.6.46.254:36175)
0810 14:32:19.048959 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:19.048964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:19.049637 (+ 673us) server_negotiation.cc:407] Connection header received
0810 14:32:19.049832 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:19.049838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:19.049919 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:19.050045 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:19.051123 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.052026 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.053883 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.054428 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.054540 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.056683 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:19.056688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:19.056691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:19.056721 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:19.059932 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:19.060866 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:19.060871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:19.060874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:19.061024 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:19.061461 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:19.061466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:19.061470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:19.061825 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:19.061986 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:19.062510 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 14:32:19.062680 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:32:19.063385 19708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.048907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36175 (local address 127.6.46.195:60215)
0810 14:32:19.049527 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:19.049548 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:19.049699 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:19.050070 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:19.050073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:19.050569 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:19.050981 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.050995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.052198 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.052202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.053744 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.053761 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.054619 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.054626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.054774 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.055342 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:19.055362 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:19.056501 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:19.060079 (+ 3578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:19.060085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:19.060089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:19.060735 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:19.061149 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:19.061159 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:19.061162 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:19.061335 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:19.061962 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:19.061967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:19.062223 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 14:32:19.062402 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 14:32:19.062715 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":346,"threads_started":1}
I20260810 14:32:19.064431 19557 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:39551
I20260810 14:32:19.064493 19773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:39551 every 8 connection(s)
I20260810 14:32:19.065935 19557 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:32:19.067219 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 19557
I20260810 14:32:19.067433 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:32:19.067926 6331 external_mini_cluster.cc:1428] Setting key d2f965d0a12440ae8e326e0e6f45d6da
I20260810 14:32:19.068930 19777 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36175
I20260810 14:32:19.069185 19777 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:19.069895 19777 heartbeater.cc:511] Master 127.6.46.254:36175 requested a full tablet report, sending...
I20260810 14:32:19.071202 18992 ts_manager.cc:194] Registered new tserver with Master: 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195:39551)
I20260810 14:32:19.072131 18992 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:35119
I20260810 14:32:19.082650 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:32:19.082770 19319 heartbeater.cc:503] Master 127.6.46.254:36175 was elected leader, sending a full tablet report...
Aug 10 14:32:19 dist-test-slave-q9h9 krb5kdc[18917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 14:32:19 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:19 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:32:19.136045 19143 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:19.126191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50818 (local address 127.6.46.254:36175)
0810 14:32:19.126411 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:19.126418 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:19.126583 (+ 165us) server_negotiation.cc:407] Connection header received
0810 14:32:19.126786 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:19.126790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:19.126870 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:19.126957 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:19.128314 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.129035 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.130302 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.130716 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.130830 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.135824 (+ 4994us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50818: BlockingRecv error: recv got EOF from 127.0.0.1:50818 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 14:32:19 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372339, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
I20260810 14:32:19.152258 19143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.139297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50828 (local address 127.6.46.254:36175)
0810 14:32:19.139481 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:19.139487 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:19.139815 (+ 328us) server_negotiation.cc:407] Connection header received
0810 14:32:19.139977 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:19.139982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:19.140049 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:19.140149 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:19.141239 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.141995 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.143199 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.143676 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.143794 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.147320 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:19.147325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:19.147328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:19.147362 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:19.149756 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:19.150420 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:19.150426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:19.150428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:19.150548 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:19.150898 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:19.150902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:19.150906 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:19.151186 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:19.151442 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:19.151702 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 14:32:19.151912 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:32:19.156138 18992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:32:19.162870 18992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:32:19.184983 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.174527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.6.46.195:39551)
0810 14:32:19.176711 (+ 2184us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:19.176720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:19.176752 (+ 32us) server_negotiation.cc:407] Connection header received
0810 14:32:19.176858 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:19.176865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:19.177096 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:19.177304 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:19.178288 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.179729 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.182321 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.183557 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.183684 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.183918 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:19.184487 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 14:32:19.184657 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2000,"thread_start_us":1813,"threads_started":1}
I20260810 14:32:19.185034 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.174174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:39551 (local address 127.0.0.1:49042)
0810 14:32:19.175974 (+ 1800us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:19.175991 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:19.176100 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:19.177649 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:19.177655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:19.177685 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:19.178057 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.178071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.179874 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.179880 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.182152 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.182164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.184048 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.184055 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.184227 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.184286 (+ 59us) client_negotiation.cc:769] Sending connection context
0810 14:32:19.184415 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 14:32:19.184582 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1685,"mutex_wait_us":576,"thread_start_us":1008,"threads_started":1}
I20260810 14:32:19.185570 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.176348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:35025 (local address 127.0.0.1:51728)
0810 14:32:19.176920 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:19.176939 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:19.177056 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:19.178157 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:19.178162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:19.178187 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:19.178499 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.178508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.180120 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.180126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.182636 (+ 2510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.182655 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.184783 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.184790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.184953 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.184991 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 14:32:19.185062 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 14:32:19.185167 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":77,"thread_start_us":754,"threads_started":1}
I20260810 14:32:19.185674 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.173995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:37085 (local address 127.0.0.1:60276)
0810 14:32:19.175077 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:19.175123 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:19.175284 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:19.178725 (+ 3441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:19.178729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:19.178754 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:19.179108 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.179120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.180767 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.180772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.182955 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.182965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.184836 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.184842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.185014 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.185050 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 14:32:19.185149 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 14:32:19.185273 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":24,"thread_start_us":406,"threads_started":1}
I20260810 14:32:19.186434 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.174511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60276 (local address 127.6.46.194:37085)
0810 14:32:19.178144 (+ 3633us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:19.178152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:19.178177 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:32:19.178270 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:19.178277 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:19.178496 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:19.178727 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:19.179291 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.180656 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.183083 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.184352 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.184474 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.184678 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:19.186070 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0810 14:32:19.186239 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3447,"thread_start_us":3249,"threads_started":1}
I20260810 14:32:19.186627 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.176931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51728 (local address 127.6.46.193:35025)
0810 14:32:19.177523 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:19.177532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:19.177560 (+ 28us) server_negotiation.cc:407] Connection header received
0810 14:32:19.177666 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:19.177674 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:19.177899 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:19.178161 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:19.178641 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.179968 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.183143 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.184541 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.185896 (+ 1355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.186109 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:19.186178 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 14:32:19.186340 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":612,"threads_started":1}
I20260810 14:32:19.189445 19430 tablet_service.cc:1467] Processing CreateTablet for tablet 1560c7b85fa246109b8a61fe1447d73e (DEFAULT_TABLE table=test-table [id=378c50447b4c46f687ad34bbeab36b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:19.189472 19667 tablet_service.cc:1467] Processing CreateTablet for tablet 1560c7b85fa246109b8a61fe1447d73e (DEFAULT_TABLE table=test-table [id=378c50447b4c46f687ad34bbeab36b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:19.190667 19430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1560c7b85fa246109b8a61fe1447d73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:19.190423 19214 tablet_service.cc:1467] Processing CreateTablet for tablet 1560c7b85fa246109b8a61fe1447d73e (DEFAULT_TABLE table=test-table [id=378c50447b4c46f687ad34bbeab36b45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:19.190953 19667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1560c7b85fa246109b8a61fe1447d73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:19.191658 19214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1560c7b85fa246109b8a61fe1447d73e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:19.201508 19827 tablet_bootstrap.cc:492] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954: Bootstrap starting.
I20260810 14:32:19.204831 19827 tablet_bootstrap.cc:654] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:19.204890 19828 tablet_bootstrap.cc:492] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020: Bootstrap starting.
I20260810 14:32:19.207367 19827 log.cc:824] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:19.208359 19828 tablet_bootstrap.cc:654] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:19.209576 19828 log.cc:824] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:19.208652 19829 tablet_bootstrap.cc:492] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2: Bootstrap starting.
I20260810 14:32:19.210502 19827 tablet_bootstrap.cc:492] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954: No bootstrap required, opened a new log
I20260810 14:32:19.210744 19827 ts_tablet_manager.cc:1397] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.005s
I20260810 14:32:19.212558 19828 tablet_bootstrap.cc:492] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020: No bootstrap required, opened a new log
I20260810 14:32:19.212589 19829 tablet_bootstrap.cc:654] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:19.212809 19828 ts_tablet_manager.cc:1397] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260810 14:32:19.213635 19829 log.cc:824] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:19.219992 19829 tablet_bootstrap.cc:492] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2: No bootstrap required, opened a new log
I20260810 14:32:19.220278 19829 ts_tablet_manager.cc:1397] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260810 14:32:19.221292 19827 raft_consensus.cc:348] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } }
I20260810 14:32:19.221719 19827 raft_consensus.cc:374] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:19.221813 19827 raft_consensus.cc:729] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92948900f73475b81fd739707311954, State: Initialized, Role: FOLLOWER
I20260810 14:32:19.222476 19827 consensus_queue.cc:260] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } }
I20260810 14:32:19.222808 19828 raft_consensus.cc:348] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } }
I20260810 14:32:19.223263 19828 raft_consensus.cc:374] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:19.223369 19828 raft_consensus.cc:729] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6521c3bfbdbf4fc68aaad46b613e6020, State: Initialized, Role: FOLLOWER
I20260810 14:32:19.223763 19552 heartbeater.cc:503] Master 127.6.46.254:36175 was elected leader, sending a full tablet report...
I20260810 14:32:19.223999 19828 consensus_queue.cc:260] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } }
I20260810 14:32:19.224298 19827 ts_tablet_manager.cc:1428] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260810 14:32:19.225755 19828 ts_tablet_manager.cc:1428] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
I20260810 14:32:19.226373 19777 heartbeater.cc:503] Master 127.6.46.254:36175 was elected leader, sending a full tablet report...
I20260810 14:32:19.229799 19829 raft_consensus.cc:348] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } }
I20260810 14:32:19.230134 19829 raft_consensus.cc:374] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:19.230230 19829 raft_consensus.cc:729] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ab8c71129b4a3aaacbadf8447307a2, State: Initialized, Role: FOLLOWER
I20260810 14:32:19.230717 19829 consensus_queue.cc:260] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } }
I20260810 14:32:19.231841 19829 ts_tablet_manager.cc:1428] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
W20260810 14:32:19.318279 19779 tablet.cc:2369] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:32:19.321141 19554 tablet.cc:2369] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:32:19.326635 19321 tablet.cc:2369] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:32:19.670507 19838 raft_consensus.cc:482] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:32:19.670845 19838 raft_consensus.cc:504] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } }
I20260810 14:32:19.672626 19838 leader_election.cc:288] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195:39551), 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193:35025)
I20260810 14:32:19.678861 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.673360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:40697 (local address 127.6.46.193:35025)
0810 14:32:19.673583 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:19.673589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:19.673745 (+ 156us) server_negotiation.cc:407] Connection header received
0810 14:32:19.673959 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:19.673963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:19.674058 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:19.674138 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:19.674691 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.675403 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.677143 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.677899 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.678067 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.678143 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:19.678496 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 14:32:19.678627 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:32:19.678965 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.673083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:49199 (local address 127.6.46.195:39551)
0810 14:32:19.673283 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:19.673290 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:19.674000 (+ 710us) server_negotiation.cc:407] Connection header received
0810 14:32:19.674194 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:19.674198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:19.674262 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:19.674336 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:19.674905 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.675574 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.677293 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.678064 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.678212 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.678284 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:19.678589 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 14:32:19.678728 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:32:19.679064 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.673006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:39551 (local address 127.6.46.194:49199)
0810 14:32:19.673888 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:19.673902 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:19.674059 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:19.674373 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:19.674377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:19.674406 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:19.674764 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.674774 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.675707 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.675712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.677136 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.677156 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.678273 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.678279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.678446 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.678469 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:32:19.678547 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:32:19.678662 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":22,"thread_start_us":639,"threads_started":1}
I20260810 14:32:19.679370 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.672901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:35025 (local address 127.6.46.194:40697)
0810 14:32:19.673647 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:19.673663 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:19.673824 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:19.674171 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:19.674176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:19.674201 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:19.674551 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.674563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.675532 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.675537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.677000 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.677012 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.678133 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.678139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.678308 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.678374 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 14:32:19.678456 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 14:32:19.678576 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"mutex_wait_us":36,"thread_start_us":379,"threads_started":1}
I20260810 14:32:19.679903 19705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1560c7b85fa246109b8a61fe1447d73e" candidate_uuid: "d92948900f73475b81fd739707311954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" is_pre_election: true
I20260810 14:32:19.680190 19240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1560c7b85fa246109b8a61fe1447d73e" candidate_uuid: "d92948900f73475b81fd739707311954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ab8c71129b4a3aaacbadf8447307a2" is_pre_election: true
I20260810 14:32:19.680305 19705 raft_consensus.cc:2454] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92948900f73475b81fd739707311954 in term 0.
I20260810 14:32:19.680619 19240 raft_consensus.cc:2454] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92948900f73475b81fd739707311954 in term 0.
I20260810 14:32:19.680864 19359 leader_election.cc:302] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6521c3bfbdbf4fc68aaad46b613e6020, d92948900f73475b81fd739707311954; no voters:
I20260810 14:32:19.681221 19838 raft_consensus.cc:2781] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:32:19.681373 19838 raft_consensus.cc:482] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:32:19.681464 19838 raft_consensus.cc:3037] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:19.683655 19838 raft_consensus.cc:504] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } }
I20260810 14:32:19.684185 19838 leader_election.cc:288] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [CANDIDATE]: Term 1 election: Requested vote from peers 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195:39551), 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193:35025)
I20260810 14:32:19.684571 19705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1560c7b85fa246109b8a61fe1447d73e" candidate_uuid: "d92948900f73475b81fd739707311954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6521c3bfbdbf4fc68aaad46b613e6020"
I20260810 14:32:19.684662 19240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1560c7b85fa246109b8a61fe1447d73e" candidate_uuid: "d92948900f73475b81fd739707311954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ab8c71129b4a3aaacbadf8447307a2"
I20260810 14:32:19.684746 19705 raft_consensus.cc:3037] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:19.684852 19240 raft_consensus.cc:3037] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:19.687024 19705 raft_consensus.cc:2454] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92948900f73475b81fd739707311954 in term 1.
I20260810 14:32:19.687186 19240 raft_consensus.cc:2454] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92948900f73475b81fd739707311954 in term 1.
I20260810 14:32:19.687389 19359 leader_election.cc:302] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6521c3bfbdbf4fc68aaad46b613e6020, d92948900f73475b81fd739707311954; no voters:
I20260810 14:32:19.687671 19838 raft_consensus.cc:2781] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:19.688067 19838 raft_consensus.cc:686] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [term 1 LEADER]: Becoming Leader. State: Replica: d92948900f73475b81fd739707311954, State: Running, Role: LEADER
I20260810 14:32:19.688561 19838 consensus_queue.cc:237] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } }
I20260810 14:32:19.692657 18992 catalog_manager.cc:5495] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 reported cstate change: term changed from 0 to 1, leader changed from <none> to d92948900f73475b81fd739707311954 (127.6.46.194). New cstate: current_term: 1 leader_uuid: "d92948900f73475b81fd739707311954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20260810 14:32:19.746986 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.741533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60288 (local address 127.6.46.194:37085)
0810 14:32:19.742189 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:19.742194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:19.742211 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:32:19.742443 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:19.742447 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:19.742544 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:19.742677 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:32:19.743068 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.743792 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.745111 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.745519 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.745608 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.745945 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:32:19.746508 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:32:19.746607 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:19.746696 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 14:32:19.746761 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":409,"threads_started":1}
I20260810 14:32:19.754769 19705 raft_consensus.cc:1261] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Refusing update from remote peer d92948900f73475b81fd739707311954: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:19.754767 19240 raft_consensus.cc:1261] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Refusing update from remote peer d92948900f73475b81fd739707311954: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:19.755852 19848 consensus_queue.cc:1035] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:19.756233 19838 consensus_queue.cc:1035] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:19.767956 19867 mvcc.cc:205] Tried to move back new op lower bound from 7316981103625084928 to 7316981103367786496. Current Snapshot: MvccSnapshot[applied={T|T < 7316981103625084928}]
I20260810 14:32:19.780114 18995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:19.780352 18995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:19.783695 18995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1: Sending DeleteTablet for 3 replicas of tablet 1560c7b85fa246109b8a61fe1447d73e
I20260810 14:32:19.784857 19430 tablet_service.cc:1514] Processing DeleteTablet for tablet 1560c7b85fa246109b8a61fe1447d73e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:19 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:60276
I20260810 14:32:19.785110 19214 tablet_service.cc:1514] Processing DeleteTablet for tablet 1560c7b85fa246109b8a61fe1447d73e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:19 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:51728
I20260810 14:32:19.785800 19667 tablet_service.cc:1514] Processing DeleteTablet for tablet 1560c7b85fa246109b8a61fe1447d73e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:19 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:49042
I20260810 14:32:19.785955 18995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:32:19.786618 18995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:32:19.790529 19883 tablet_replica.cc:330] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020: stopping tablet replica
I20260810 14:32:19.790995 19883 raft_consensus.cc:2229] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:19.791515 19883 raft_consensus.cc:2258] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:19.795668 19883 ts_tablet_manager.cc:1905] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:19.802908 19883 ts_tablet_manager.cc:1918] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:19.803223 19883 log.cc:1196] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/1560c7b85fa246109b8a61fe1447d73e
I20260810 14:32:19.803898 19883 ts_tablet_manager.cc:1939] T 1560c7b85fa246109b8a61fe1447d73e P 6521c3bfbdbf4fc68aaad46b613e6020: Deleting consensus metadata
I20260810 14:32:19.805866 18977 catalog_manager.cc:4841] TS 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195:39551): tablet 1560c7b85fa246109b8a61fe1447d73e (table test-table [id=378c50447b4c46f687ad34bbeab36b45]) successfully deleted
I20260810 14:32:19.806882 19880 tablet_replica.cc:330] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954: stopping tablet replica
I20260810 14:32:19.807018 19882 tablet_replica.cc:330] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2: stopping tablet replica
I20260810 14:32:19.807325 19880 raft_consensus.cc:2229] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:32:19.807569 19882 raft_consensus.cc:2229] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:19.808017 19880 raft_consensus.cc:2258] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:19.808125 19882 raft_consensus.cc:2258] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:19.809535 19667 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6ac5ef2679469fb6f0d8f13b04252a (DEFAULT_TABLE table=test-table [id=7ecef1b4c88243d48d4792b7dd1df4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:19.809944 19880 ts_tablet_manager.cc:1905] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:19.809993 19667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6ac5ef2679469fb6f0d8f13b04252a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:19.810230 19882 ts_tablet_manager.cc:1905] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:19.812067 19214 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6ac5ef2679469fb6f0d8f13b04252a (DEFAULT_TABLE table=test-table [id=7ecef1b4c88243d48d4792b7dd1df4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:19.812506 19214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6ac5ef2679469fb6f0d8f13b04252a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:19.817364 19882 ts_tablet_manager.cc:1918] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:19.817386 19880 ts_tablet_manager.cc:1918] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:19.817572 19880 log.cc:1196] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/1560c7b85fa246109b8a61fe1447d73e
I20260810 14:32:19.817601 19882 log.cc:1196] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/1560c7b85fa246109b8a61fe1447d73e
I20260810 14:32:19.818151 19880 ts_tablet_manager.cc:1939] T 1560c7b85fa246109b8a61fe1447d73e P d92948900f73475b81fd739707311954: Deleting consensus metadata
I20260810 14:32:19.818316 19882 ts_tablet_manager.cc:1939] T 1560c7b85fa246109b8a61fe1447d73e P 68ab8c71129b4a3aaacbadf8447307a2: Deleting consensus metadata
I20260810 14:32:19.819988 18972 catalog_manager.cc:4841] TS d92948900f73475b81fd739707311954 (127.6.46.194:37085): tablet 1560c7b85fa246109b8a61fe1447d73e (table test-table [id=378c50447b4c46f687ad34bbeab36b45]) successfully deleted
I20260810 14:32:19.820001 18976 catalog_manager.cc:4841] TS 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193:35025): tablet 1560c7b85fa246109b8a61fe1447d73e (table test-table [id=378c50447b4c46f687ad34bbeab36b45]) successfully deleted
I20260810 14:32:19.820994 19430 tablet_service.cc:1467] Processing CreateTablet for tablet 4e6ac5ef2679469fb6f0d8f13b04252a (DEFAULT_TABLE table=test-table [id=7ecef1b4c88243d48d4792b7dd1df4a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:19.821521 19430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e6ac5ef2679469fb6f0d8f13b04252a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:19.827742 19889 tablet_bootstrap.cc:492] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020: Bootstrap starting.
I20260810 14:32:19.831202 19889 tablet_bootstrap.cc:654] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:19.834383 19888 tablet_bootstrap.cc:492] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2: Bootstrap starting.
I20260810 14:32:19.835049 19890 tablet_bootstrap.cc:492] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954: Bootstrap starting.
I20260810 14:32:19.836253 19888 tablet_bootstrap.cc:654] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:19.837069 19890 tablet_bootstrap.cc:654] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:19.837548 19889 tablet_bootstrap.cc:492] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020: No bootstrap required, opened a new log
I20260810 14:32:19.837752 19889 ts_tablet_manager.cc:1397] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260810 14:32:19.838593 19889 raft_consensus.cc:348] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.838927 19889 raft_consensus.cc:374] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:19.839083 19889 raft_consensus.cc:729] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6521c3bfbdbf4fc68aaad46b613e6020, State: Initialized, Role: FOLLOWER
I20260810 14:32:19.839402 19889 consensus_queue.cc:260] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.840641 19889 ts_tablet_manager.cc:1428] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260810 14:32:19.846357 19888 tablet_bootstrap.cc:492] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2: No bootstrap required, opened a new log
I20260810 14:32:19.846511 19888 ts_tablet_manager.cc:1397] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20260810 14:32:19.846817 19890 tablet_bootstrap.cc:492] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954: No bootstrap required, opened a new log
I20260810 14:32:19.846964 19890 ts_tablet_manager.cc:1397] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20260810 14:32:19.847260 19888 raft_consensus.cc:348] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.847550 19888 raft_consensus.cc:374] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:19.847661 19888 raft_consensus.cc:729] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ab8c71129b4a3aaacbadf8447307a2, State: Initialized, Role: FOLLOWER
I20260810 14:32:19.847632 19890 raft_consensus.cc:348] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.847878 19890 raft_consensus.cc:374] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:19.847980 19890 raft_consensus.cc:729] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92948900f73475b81fd739707311954, State: Initialized, Role: FOLLOWER
I20260810 14:32:19.847890 19888 consensus_queue.cc:260] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.848469 19888 ts_tablet_manager.cc:1428] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:32:19.848306 19890 consensus_queue.cc:260] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.848997 19890 ts_tablet_manager.cc:1428] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:32:19.854375 19863 raft_consensus.cc:482] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:32:19.854583 19863 raft_consensus.cc:504] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.856314 19863 leader_election.cc:288] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193:35025), d92948900f73475b81fd739707311954 (127.6.46.194:37085)
I20260810 14:32:19.863566 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.856857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:39203 (local address 127.6.46.193:35025)
0810 14:32:19.857016 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:19.857022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:19.857158 (+ 136us) server_negotiation.cc:407] Connection header received
0810 14:32:19.857444 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:19.857451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:19.857538 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:19.857604 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:19.858343 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.859255 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.861311 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.862624 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.862777 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.862852 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:19.863193 (+ 341us) server_negotiation.cc:299] Negotiation successful
0810 14:32:19.863320 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:32:19.863571 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.856458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:35025 (local address 127.6.46.195:39203)
0810 14:32:19.857055 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:19.857074 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:19.857276 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:19.857797 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:19.857802 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:19.857828 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:19.858178 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.858188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.859403 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.859406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.861136 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.861161 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.862814 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.862821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.862983 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.863018 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 14:32:19.863157 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 14:32:19.863286 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":805,"threads_started":1}
I20260810 14:32:19.863845 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.857218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:36507 (local address 127.6.46.194:37085)
0810 14:32:19.857363 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:19.857370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:19.857874 (+ 504us) server_negotiation.cc:407] Connection header received
0810 14:32:19.858070 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:19.858075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:19.858144 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:19.858241 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:19.858835 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.859974 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.861906 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.862830 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.862999 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.863066 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:19.863453 (+ 387us) server_negotiation.cc:299] Negotiation successful
0810 14:32:19.863600 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 14:32:19.864504 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.856700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:37085 (local address 127.6.46.195:36507)
0810 14:32:19.857782 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:19.857798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:19.857935 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:19.858268 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:19.858272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:19.858298 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:19.858673 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.858686 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.860116 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.860122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.861772 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.861783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.863057 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.863064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.863260 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.863295 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 14:32:19.864000 (+ 705us) client_negotiation.cc:240] Negotiation successful
0810 14:32:19.864135 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"spinlock_wait_cycles":28416,"thread_start_us":811,"threads_started":1}
I20260810 14:32:19.864455 19240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6ac5ef2679469fb6f0d8f13b04252a" candidate_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ab8c71129b4a3aaacbadf8447307a2" is_pre_election: true
I20260810 14:32:19.864661 19240 raft_consensus.cc:2454] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6521c3bfbdbf4fc68aaad46b613e6020 in term 0.
I20260810 14:32:19.865185 19583 leader_election.cc:302] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6521c3bfbdbf4fc68aaad46b613e6020, 68ab8c71129b4a3aaacbadf8447307a2; no voters:
I20260810 14:32:19.865578 19863 raft_consensus.cc:2781] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:32:19.865464 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6ac5ef2679469fb6f0d8f13b04252a" candidate_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92948900f73475b81fd739707311954" is_pre_election: true
I20260810 14:32:19.865691 19863 raft_consensus.cc:482] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:32:19.865782 19863 raft_consensus.cc:3037] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:19.865880 19471 raft_consensus.cc:2454] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6521c3bfbdbf4fc68aaad46b613e6020 in term 0.
I20260810 14:32:19.868041 19863 raft_consensus.cc:504] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.868608 19863 leader_election.cc:288] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [CANDIDATE]: Term 1 election: Requested vote from peers 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193:35025), d92948900f73475b81fd739707311954 (127.6.46.194:37085)
I20260810 14:32:19.868975 19240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6ac5ef2679469fb6f0d8f13b04252a" candidate_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ab8c71129b4a3aaacbadf8447307a2"
I20260810 14:32:19.869060 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e6ac5ef2679469fb6f0d8f13b04252a" candidate_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92948900f73475b81fd739707311954"
I20260810 14:32:19.869187 19240 raft_consensus.cc:3037] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:19.869233 19471 raft_consensus.cc:3037] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:19.871977 19240 raft_consensus.cc:2454] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6521c3bfbdbf4fc68aaad46b613e6020 in term 1.
I20260810 14:32:19.872345 19583 leader_election.cc:302] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6521c3bfbdbf4fc68aaad46b613e6020, 68ab8c71129b4a3aaacbadf8447307a2; no voters:
I20260810 14:32:19.872386 19471 raft_consensus.cc:2454] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6521c3bfbdbf4fc68aaad46b613e6020 in term 1.
I20260810 14:32:19.872637 19863 raft_consensus.cc:2781] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:19.873072 19863 raft_consensus.cc:686] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 LEADER]: Becoming Leader. State: Replica: 6521c3bfbdbf4fc68aaad46b613e6020, State: Running, Role: LEADER
I20260810 14:32:19.873462 19863 consensus_queue.cc:237] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.876857 18995 catalog_manager.cc:5495] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195). New cstate: current_term: 1 leader_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } health_report { overall_health: HEALTHY } } }
Aug 10 14:32:19 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372337, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.254@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
I20260810 14:32:19.946762 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.932237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36175 (local address 127.0.0.1:50844)
0810 14:32:19.932993 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:19.933013 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:19.933151 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:19.933447 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:19.933455 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:19.934158 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:19.934394 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.934404 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.935468 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.935472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.936460 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:19.936473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.937248 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.937255 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:19.937403 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.938132 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:19.938170 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:19.940404 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:19.944023 (+ 3619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:19.944034 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:19.944039 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:19.944529 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:19.944895 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:19.944898 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:19.944901 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:19.945036 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:19.945653 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:19.945661 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:19.945857 (+ 196us) client_negotiation.cc:769] Sending connection context
0810 14:32:19.946063 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 14:32:19.946323 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":505,"threads_started":1}
I20260810 14:32:19.946781 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:19.932230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50844 (local address 127.6.46.254:36175)
0810 14:32:19.932868 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:19.932873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:19.933089 (+ 216us) server_negotiation.cc:407] Connection header received
0810 14:32:19.933256 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:19.933261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:19.933316 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:19.933402 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:19.934558 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.935345 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.936590 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:19.937069 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:19.937176 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:19.940620 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:19.940628 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:19.940633 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:19.940670 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:19.943907 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:19.944630 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:19.944636 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:19.944639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:19.944794 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:19.945136 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:19.945140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:19.945144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:19.945514 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:19.945658 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:19.946229 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 14:32:19.946420 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":277,"threads_started":1}
I20260810 14:32:19.952222 18995 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:50844:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 14:32:19.953212 18995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:32:19.960085 19214 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6175c412a24a0b9855add33113c2ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dac82c238aa049e296a4e3327d721b3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:32:19.960223 19430 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6175c412a24a0b9855add33113c2ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dac82c238aa049e296a4e3327d721b3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:32:19.960340 19667 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6175c412a24a0b9855add33113c2ae (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=dac82c238aa049e296a4e3327d721b3c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 14:32:19.960668 19214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6175c412a24a0b9855add33113c2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:19.960791 19430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6175c412a24a0b9855add33113c2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:19.960800 19667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6175c412a24a0b9855add33113c2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:19.967748 19890 tablet_bootstrap.cc:492] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954: Bootstrap starting.
I20260810 14:32:19.968638 19888 tablet_bootstrap.cc:492] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2: Bootstrap starting.
I20260810 14:32:19.969369 19890 tablet_bootstrap.cc:654] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:19.970264 19888 tablet_bootstrap.cc:654] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:19.970355 19889 tablet_bootstrap.cc:492] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020: Bootstrap starting.
I20260810 14:32:19.971915 19889 tablet_bootstrap.cc:654] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:19.973011 19888 tablet_bootstrap.cc:492] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2: No bootstrap required, opened a new log
I20260810 14:32:19.973171 19888 ts_tablet_manager.cc:1397] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 14:32:19.973882 19888 raft_consensus.cc:348] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.974164 19888 raft_consensus.cc:374] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:19.974246 19888 raft_consensus.cc:729] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ab8c71129b4a3aaacbadf8447307a2, State: Initialized, Role: FOLLOWER
I20260810 14:32:19.974217 19890 tablet_bootstrap.cc:492] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954: No bootstrap required, opened a new log
I20260810 14:32:19.974431 19890 ts_tablet_manager.cc:1397] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260810 14:32:19.974476 19888 consensus_queue.cc:260] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.975131 19890 raft_consensus.cc:348] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.975200 19864 tablet_replica.cc:439] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } }
I20260810 14:32:19.975483 19890 raft_consensus.cc:374] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:19.975517 19864 tablet_replica.cc:442] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:32:19.975574 19890 raft_consensus.cc:729] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92948900f73475b81fd739707311954, State: Initialized, Role: FOLLOWER
I20260810 14:32:19.975835 19890 consensus_queue.cc:260] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.976337 19890 ts_tablet_manager.cc:1428] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 14:32:19.976246 19872 tablet_replica.cc:439] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } }
I20260810 14:32:19.976595 19872 tablet_replica.cc:442] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:32:19.977615 19888 ts_tablet_manager.cc:1428] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 14:32:19.977777 19889 tablet_bootstrap.cc:492] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020: No bootstrap required, opened a new log
I20260810 14:32:19.977928 19889 ts_tablet_manager.cc:1397] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260810 14:32:19.979004 19889 raft_consensus.cc:348] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.979420 19889 raft_consensus.cc:374] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:19.979506 19889 raft_consensus.cc:729] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6521c3bfbdbf4fc68aaad46b613e6020, State: Initialized, Role: FOLLOWER
I20260810 14:32:19.979724 19889 consensus_queue.cc:260] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:19.980044 19863 tablet_replica.cc:439] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } }
I20260810 14:32:19.980266 19863 tablet_replica.cc:442] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:32:19.980602 19889 ts_tablet_manager.cc:1428] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260810 14:32:20.175422 19864 raft_consensus.cc:482] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:32:20.175673 19864 raft_consensus.cc:504] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.177469 19864 leader_election.cc:288] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92948900f73475b81fd739707311954 (127.6.46.194:37085), 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195:39551)
I20260810 14:32:20.185535 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:20.178067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:60335 (local address 127.6.46.194:37085)
0810 14:32:20.178250 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:20.178256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:20.178562 (+ 306us) server_negotiation.cc:407] Connection header received
0810 14:32:20.178863 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:20.178870 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:20.178973 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:20.179050 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:20.179846 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.180783 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.182961 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.184221 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.184375 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.184449 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:20.185116 (+ 667us) server_negotiation.cc:299] Negotiation successful
0810 14:32:20.185247 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:32:20.185583 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:20.177791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:37085 (local address 127.6.46.193:60335)
0810 14:32:20.178443 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:20.178464 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:20.178693 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:20.179280 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:20.179284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:20.179313 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:20.179698 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:20.179712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.180899 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.180904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:20.182766 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:20.182784 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.184419 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.184425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:20.184956 (+ 531us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.184983 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 14:32:20.185066 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 14:32:20.185190 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":3,"thread_start_us":1140,"threads_started":1}
I20260810 14:32:20.185581 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:20.178540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:57509 (local address 127.6.46.195:39551)
0810 14:32:20.178726 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:20.178732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:20.179401 (+ 669us) server_negotiation.cc:407] Connection header received
0810 14:32:20.179727 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:20.179734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:20.179812 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:20.179894 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:20.180487 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.181598 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.183628 (+ 2030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.184447 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.184593 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.184661 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:20.185171 (+ 510us) server_negotiation.cc:299] Negotiation successful
0810 14:32:20.185311 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 14:32:20.186188 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6175c412a24a0b9855add33113c2ae" candidate_uuid: "68ab8c71129b4a3aaacbadf8447307a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92948900f73475b81fd739707311954" is_pre_election: true
I20260810 14:32:20.186452 19920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:20.177786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:39551 (local address 127.6.46.193:57509)
0810 14:32:20.179281 (+ 1495us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:20.179298 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:20.179514 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:20.179919 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:20.179923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:20.179948 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:20.180317 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:20.180330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.181743 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.181749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:20.183466 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:20.183478 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.184707 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.184714 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:20.184962 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.184999 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 14:32:20.185976 (+ 977us) client_negotiation.cc:240] Negotiation successful
0810 14:32:20.186111 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1370,"mutex_wait_us":67,"thread_start_us":334,"threads_started":1}
I20260810 14:32:20.186497 19471 raft_consensus.cc:2454] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ab8c71129b4a3aaacbadf8447307a2 in term 0.
I20260810 14:32:20.186980 19130 leader_election.cc:302] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68ab8c71129b4a3aaacbadf8447307a2, d92948900f73475b81fd739707311954; no voters:
I20260810 14:32:20.187162 19705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6175c412a24a0b9855add33113c2ae" candidate_uuid: "68ab8c71129b4a3aaacbadf8447307a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" is_pre_election: true
I20260810 14:32:20.187377 19864 raft_consensus.cc:2781] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:32:20.187389 19705 raft_consensus.cc:2454] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ab8c71129b4a3aaacbadf8447307a2 in term 0.
I20260810 14:32:20.187485 19864 raft_consensus.cc:482] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:32:20.187570 19864 raft_consensus.cc:3037] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:20.190572 19864 raft_consensus.cc:504] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.191207 19864 leader_election.cc:288] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [CANDIDATE]: Term 1 election: Requested vote from peers d92948900f73475b81fd739707311954 (127.6.46.194:37085), 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195:39551)
I20260810 14:32:20.191635 19705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6175c412a24a0b9855add33113c2ae" candidate_uuid: "68ab8c71129b4a3aaacbadf8447307a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6521c3bfbdbf4fc68aaad46b613e6020"
I20260810 14:32:20.191642 19471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6175c412a24a0b9855add33113c2ae" candidate_uuid: "68ab8c71129b4a3aaacbadf8447307a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92948900f73475b81fd739707311954"
I20260810 14:32:20.191811 19705 raft_consensus.cc:3037] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:20.191864 19471 raft_consensus.cc:3037] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:20.193778 19705 raft_consensus.cc:2454] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ab8c71129b4a3aaacbadf8447307a2 in term 1.
I20260810 14:32:20.194165 19135 leader_election.cc:302] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6521c3bfbdbf4fc68aaad46b613e6020, 68ab8c71129b4a3aaacbadf8447307a2; no voters:
I20260810 14:32:20.194414 19864 raft_consensus.cc:2781] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:20.194864 19864 raft_consensus.cc:686] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 LEADER]: Becoming Leader. State: Replica: 68ab8c71129b4a3aaacbadf8447307a2, State: Running, Role: LEADER
I20260810 14:32:20.194976 19471 raft_consensus.cc:2454] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ab8c71129b4a3aaacbadf8447307a2 in term 1.
I20260810 14:32:20.195257 19864 consensus_queue.cc:237] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.197347 19922 tablet_replica.cc:439] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2: TxnStatusTablet state changed. Reason: New leader 68ab8c71129b4a3aaacbadf8447307a2. Latest consensus state: current_term: 1 leader_uuid: "68ab8c71129b4a3aaacbadf8447307a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } }
I20260810 14:32:20.197809 19922 tablet_replica.cc:442] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:32:20.198793 19926 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 14:32:20.199018 19926 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 14:32:20.199236 18995 catalog_manager.cc:5495] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193). New cstate: current_term: 1 leader_uuid: "68ab8c71129b4a3aaacbadf8447307a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } health_report { overall_health: UNKNOWN } } }
I20260810 14:32:20.218461 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:20.211376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51742 (local address 127.6.46.193:35025)
0810 14:32:20.211566 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:20.211573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:20.211790 (+ 217us) server_negotiation.cc:407] Connection header received
0810 14:32:20.212073 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:20.212078 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:20.212201 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:20.212352 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:32:20.212749 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.213901 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.215803 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.216458 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.216594 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.217116 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:32:20.217786 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:32:20.217936 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:20.218055 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 14:32:20.218145 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:32:20.218480 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:20.211498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:35025 (local address 127.0.0.1:51742)
0810 14:32:20.211702 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:20.211721 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:20.211875 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:20.212373 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:20.212379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:20.212392 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:32:20.212606 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:20.212624 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.214054 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.214061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:20.215639 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:20.215654 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.216639 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.216645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:20.216854 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.216923 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:32:20.217898 (+ 975us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:32:20.217920 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:32:20.217998 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:32:20.218182 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 14:32:20.222810 19705 raft_consensus.cc:1261] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Refusing update from remote peer 68ab8c71129b4a3aaacbadf8447307a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:20.223199 19471 raft_consensus.cc:1261] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Refusing update from remote peer 68ab8c71129b4a3aaacbadf8447307a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:20.223775 19922 consensus_queue.cc:1035] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:20.224277 19864 consensus_queue.cc:1035] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:20.228850 19863 tablet_replica.cc:439] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020: TxnStatusTablet state changed. Reason: New leader 68ab8c71129b4a3aaacbadf8447307a2. Latest consensus state: current_term: 1 leader_uuid: "68ab8c71129b4a3aaacbadf8447307a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } }
I20260810 14:32:20.229166 19863 tablet_replica.cc:442] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:32:20.231323 19866 mvcc.cc:205] Tried to move back new op lower bound from 7316981105545822208 to 7316981105443454976. Current Snapshot: MvccSnapshot[applied={T|T < 7316981105545822208 or (T in {7316981105545822208})}]
I20260810 14:32:20.232113 19863 tablet_replica.cc:439] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68ab8c71129b4a3aaacbadf8447307a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } }
I20260810 14:32:20.232388 19863 tablet_replica.cc:442] T 1c6175c412a24a0b9855add33113c2ae P 6521c3bfbdbf4fc68aaad46b613e6020: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:32:20.233695 19922 tablet_replica.cc:439] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68ab8c71129b4a3aaacbadf8447307a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } }
I20260810 14:32:20.233939 19922 tablet_replica.cc:442] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:32:20.237035 19872 tablet_replica.cc:439] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954: TxnStatusTablet state changed. Reason: New leader 68ab8c71129b4a3aaacbadf8447307a2. Latest consensus state: current_term: 1 leader_uuid: "68ab8c71129b4a3aaacbadf8447307a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } }
I20260810 14:32:20.237275 19872 tablet_replica.cc:442] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:32:20.241964 19943 tablet_replica.cc:439] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "68ab8c71129b4a3aaacbadf8447307a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } }
I20260810 14:32:20.242379 19943 tablet_replica.cc:442] T 1c6175c412a24a0b9855add33113c2ae P 68ab8c71129b4a3aaacbadf8447307a2: This TxnStatusTablet replica's current role is: LEADER
I20260810 14:32:20.245266 19872 tablet_replica.cc:439] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "68ab8c71129b4a3aaacbadf8447307a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } } }
I20260810 14:32:20.245523 19872 tablet_replica.cc:442] T 1c6175c412a24a0b9855add33113c2ae P d92948900f73475b81fd739707311954: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 14:32:20.249193 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:20.243003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.6.46.195:39551)
0810 14:32:20.243196 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:20.243202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:20.243220 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:32:20.243311 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:20.243317 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:20.243422 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:20.243574 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:32:20.244102 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.244888 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.246651 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.247291 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.247421 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.248049 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:32:20.248554 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:32:20.248662 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:20.248771 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 14:32:20.248904 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:32:20.263906 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:20.258213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:48159 (local address 127.6.46.193:35025)
0810 14:32:20.258385 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:20.258390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:20.258794 (+ 404us) server_negotiation.cc:407] Connection header received
0810 14:32:20.258962 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:20.258966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:20.259023 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:20.259090 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:32:20.259587 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.260280 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.262218 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.262696 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.262788 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.263276 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:32:20.263372 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:32:20.263434 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:20.263622 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 14:32:20.263681 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:32:20.264118 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:20.258075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:35025 (local address 127.6.46.195:48159)
0810 14:32:20.258665 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:20.258692 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:20.258857 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:20.259125 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:20.259130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:20.259143 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:32:20.259458 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:20.259470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.260403 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.260409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:20.262090 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:20.262105 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.262868 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.262874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:20.263040 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.263105 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:32:20.263464 (+ 359us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:32:20.263476 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 14:32:20.263610 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 14:32:20.263743 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":270,"threads_started":1}
I20260810 14:32:20.273063 19471 raft_consensus.cc:1261] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Refusing update from remote peer 6521c3bfbdbf4fc68aaad46b613e6020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:20.273192 19240 raft_consensus.cc:1261] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Refusing update from remote peer 6521c3bfbdbf4fc68aaad46b613e6020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:20.273737 19896 consensus_queue.cc:1035] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:20.274423 19896 consensus_queue.cc:1035] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:20.291085 19862 mvcc.cc:205] Tried to move back new op lower bound from 7316981105753239552 to 7316981104125411328. Current Snapshot: MvccSnapshot[applied={T|T < 7316981105753239552}]
I20260810 14:32:20.310699 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:20.304246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:60741 (local address 127.6.46.195:39551)
0810 14:32:20.304424 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:20.304431 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:20.305133 (+ 702us) server_negotiation.cc:407] Connection header received
0810 14:32:20.305294 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:20.305299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:20.305380 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:20.305475 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:32:20.305962 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.307147 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.308826 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.309459 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.309594 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.309988 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:32:20.310147 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:32:20.310232 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:20.310348 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 14:32:20.310423 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:32:20.310896 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:20.304326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:39551 (local address 127.6.46.193:60741)
0810 14:32:20.305042 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:20.305058 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:20.305182 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:20.305476 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:20.305479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:20.305488 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 14:32:20.305825 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:20.305835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.307386 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.307392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:20.308689 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:20.308703 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.309623 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.309628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:20.309768 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.309818 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 14:32:20.310233 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 14:32:20.310254 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 14:32:20.310330 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 14:32:20.310461 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":380,"threads_started":1}
I20260810 14:32:20.347506 18995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:20.347774 18995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:20.351073 18995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1: Sending DeleteTablet for 3 replicas of tablet 4e6ac5ef2679469fb6f0d8f13b04252a
I20260810 14:32:20.351809 19214 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e6ac5ef2679469fb6f0d8f13b04252a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:20 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:51728
I20260810 14:32:20.352180 19430 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e6ac5ef2679469fb6f0d8f13b04252a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:20 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:60276
I20260810 14:32:20.352892 19667 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e6ac5ef2679469fb6f0d8f13b04252a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:20 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:49042
I20260810 14:32:20.377313 19966 tablet_replica.cc:330] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954: stopping tablet replica
I20260810 14:32:20.377550 19966 raft_consensus.cc:2229] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:20.377784 19966 raft_consensus.cc:2258] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:20.378456 19968 tablet_replica.cc:330] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020: stopping tablet replica
I20260810 14:32:20.378669 19968 raft_consensus.cc:2229] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:32:20.379220 19968 raft_consensus.cc:2258] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:20.380398 19966 ts_tablet_manager.cc:1905] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:20.386135 19968 ts_tablet_manager.cc:1905] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:20.390401 19961 tablet_replica.cc:330] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2: stopping tablet replica
I20260810 14:32:20.390621 19961 raft_consensus.cc:2229] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:20.390910 19961 raft_consensus.cc:2258] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:20.392092 19961 ts_tablet_manager.cc:1905] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:20.400534 19966 ts_tablet_manager.cc:1918] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:32:20.400672 19966 log.cc:1196] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/4e6ac5ef2679469fb6f0d8f13b04252a
I20260810 14:32:20.401193 19966 ts_tablet_manager.cc:1939] T 4e6ac5ef2679469fb6f0d8f13b04252a P d92948900f73475b81fd739707311954: Deleting consensus metadata
I20260810 14:32:20.402453 18972 catalog_manager.cc:4841] TS d92948900f73475b81fd739707311954 (127.6.46.194:37085): tablet 4e6ac5ef2679469fb6f0d8f13b04252a (table test-table [id=7ecef1b4c88243d48d4792b7dd1df4a1]) successfully deleted
I20260810 14:32:20.403760 19961 ts_tablet_manager.cc:1918] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:32:20.403900 19961 log.cc:1196] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/4e6ac5ef2679469fb6f0d8f13b04252a
I20260810 14:32:20.404368 19961 ts_tablet_manager.cc:1939] T 4e6ac5ef2679469fb6f0d8f13b04252a P 68ab8c71129b4a3aaacbadf8447307a2: Deleting consensus metadata
I20260810 14:32:20.405417 18976 catalog_manager.cc:4841] TS 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193:35025): tablet 4e6ac5ef2679469fb6f0d8f13b04252a (table test-table [id=7ecef1b4c88243d48d4792b7dd1df4a1]) successfully deleted
I20260810 14:32:20.406070 19968 ts_tablet_manager.cc:1918] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:32:20.406234 19968 log.cc:1196] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/4e6ac5ef2679469fb6f0d8f13b04252a
I20260810 14:32:20.406632 19968 ts_tablet_manager.cc:1939] T 4e6ac5ef2679469fb6f0d8f13b04252a P 6521c3bfbdbf4fc68aaad46b613e6020: Deleting consensus metadata
I20260810 14:32:20.407655 18977 catalog_manager.cc:4841] TS 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195:39551): tablet 4e6ac5ef2679469fb6f0d8f13b04252a (table test-table [id=7ecef1b4c88243d48d4792b7dd1df4a1]) successfully deleted
Aug 10 14:32:20 dist-test-slave-q9h9 krb5kdc[18917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 14:32:20 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:20 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:32:20.437332 19909 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:20.424970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50852 (local address 127.6.46.254:36175)
0810 14:32:20.425163 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:20.425170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:20.425598 (+ 428us) server_negotiation.cc:407] Connection header received
0810 14:32:20.425804 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:20.425810 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:20.425880 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:20.425964 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:20.427256 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.427977 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.429175 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.429597 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.429719 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.437081 (+ 7362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50852: BlockingRecv error: recv got EOF from 127.0.0.1:50852 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 14:32:20 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM
I20260810 14:32:20.453368 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:20.440077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.6.46.254:36175)
0810 14:32:20.440253 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:20.440260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:20.440665 (+ 405us) server_negotiation.cc:407] Connection header received
0810 14:32:20.440801 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:20.440806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:20.440863 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:20.440939 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:20.442096 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.442880 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.444123 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.444619 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.444735 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.448269 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:20.448274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:20.448296 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:20.448337 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:20.450943 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:20.451527 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:20.451532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:20.451536 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:20.451662 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:20.452026 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:20.452031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:20.452034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:20.452293 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:20.452533 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:20.452833 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 14:32:20.453040 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 14:32:20.456688 18995 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:32:20.457297 18995 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:32:20.463984 19430 tablet_service.cc:1467] Processing CreateTablet for tablet dae070e065114dc2bf52546416835166 (DEFAULT_TABLE table=test-table [id=54a28a8660174e40a03a13b4617554fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:20.464051 19214 tablet_service.cc:1467] Processing CreateTablet for tablet dae070e065114dc2bf52546416835166 (DEFAULT_TABLE table=test-table [id=54a28a8660174e40a03a13b4617554fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:20.464438 19430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae070e065114dc2bf52546416835166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:20.464445 19214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae070e065114dc2bf52546416835166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:20.468758 19667 tablet_service.cc:1467] Processing CreateTablet for tablet dae070e065114dc2bf52546416835166 (DEFAULT_TABLE table=test-table [id=54a28a8660174e40a03a13b4617554fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:20.469177 19667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae070e065114dc2bf52546416835166. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:20.471078 19890 tablet_bootstrap.cc:492] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954: Bootstrap starting.
I20260810 14:32:20.472617 19890 tablet_bootstrap.cc:654] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:20.476615 19888 tablet_bootstrap.cc:492] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2: Bootstrap starting.
I20260810 14:32:20.478526 19888 tablet_bootstrap.cc:654] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:20.479617 19889 tablet_bootstrap.cc:492] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020: Bootstrap starting.
I20260810 14:32:20.481217 19889 tablet_bootstrap.cc:654] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:20.495416 19889 tablet_bootstrap.cc:492] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020: No bootstrap required, opened a new log
I20260810 14:32:20.495602 19889 ts_tablet_manager.cc:1397] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20260810 14:32:20.496389 19889 raft_consensus.cc:348] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.496647 19889 raft_consensus.cc:374] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:20.496743 19889 raft_consensus.cc:729] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6521c3bfbdbf4fc68aaad46b613e6020, State: Initialized, Role: FOLLOWER
I20260810 14:32:20.497005 19889 consensus_queue.cc:260] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.498595 19889 ts_tablet_manager.cc:1428] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260810 14:32:20.500403 19890 tablet_bootstrap.cc:492] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954: No bootstrap required, opened a new log
I20260810 14:32:20.500551 19890 ts_tablet_manager.cc:1397] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.000s
I20260810 14:32:20.501350 19890 raft_consensus.cc:348] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.501587 19890 raft_consensus.cc:374] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:20.501679 19890 raft_consensus.cc:729] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92948900f73475b81fd739707311954, State: Initialized, Role: FOLLOWER
I20260810 14:32:20.501883 19890 consensus_queue.cc:260] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.502507 19890 ts_tablet_manager.cc:1428] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 14:32:20.503932 19888 tablet_bootstrap.cc:492] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2: No bootstrap required, opened a new log
I20260810 14:32:20.504091 19888 ts_tablet_manager.cc:1397] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.006s
I20260810 14:32:20.504849 19888 raft_consensus.cc:348] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.505095 19888 raft_consensus.cc:374] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:20.505189 19888 raft_consensus.cc:729] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ab8c71129b4a3aaacbadf8447307a2, State: Initialized, Role: FOLLOWER
I20260810 14:32:20.505434 19888 consensus_queue.cc:260] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.506116 19888 ts_tablet_manager.cc:1428] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260810 14:32:20.712745 19872 raft_consensus.cc:482] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:32:20.713011 19872 raft_consensus.cc:504] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.713886 19872 leader_election.cc:288] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193:35025), 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195:39551)
I20260810 14:32:20.714414 19944 raft_consensus.cc:482] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:32:20.714576 19240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae070e065114dc2bf52546416835166" candidate_uuid: "d92948900f73475b81fd739707311954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ab8c71129b4a3aaacbadf8447307a2" is_pre_election: true
I20260810 14:32:20.714582 19705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae070e065114dc2bf52546416835166" candidate_uuid: "d92948900f73475b81fd739707311954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" is_pre_election: true
I20260810 14:32:20.714617 19944 raft_consensus.cc:504] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.714864 19705 raft_consensus.cc:2454] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92948900f73475b81fd739707311954 in term 0.
I20260810 14:32:20.715104 19240 raft_consensus.cc:2454] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92948900f73475b81fd739707311954 in term 0.
I20260810 14:32:20.715282 19944 leader_election.cc:288] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92948900f73475b81fd739707311954 (127.6.46.194:37085), 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195:39551)
I20260810 14:32:20.715775 19357 leader_election.cc:302] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68ab8c71129b4a3aaacbadf8447307a2, d92948900f73475b81fd739707311954; no voters:
I20260810 14:32:20.715929 19467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae070e065114dc2bf52546416835166" candidate_uuid: "68ab8c71129b4a3aaacbadf8447307a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92948900f73475b81fd739707311954" is_pre_election: true
I20260810 14:32:20.716123 19872 raft_consensus.cc:2781] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:32:20.716254 19872 raft_consensus.cc:482] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:32:20.716214 19705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae070e065114dc2bf52546416835166" candidate_uuid: "68ab8c71129b4a3aaacbadf8447307a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" is_pre_election: true
I20260810 14:32:20.716342 19872 raft_consensus.cc:3037] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:20.716411 19705 raft_consensus.cc:2454] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 68ab8c71129b4a3aaacbadf8447307a2 in term 0.
I20260810 14:32:20.716899 19135 leader_election.cc:302] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6521c3bfbdbf4fc68aaad46b613e6020, 68ab8c71129b4a3aaacbadf8447307a2; no voters:
I20260810 14:32:20.717159 19944 raft_consensus.cc:2781] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:32:20.717285 19944 raft_consensus.cc:482] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:32:20.717398 19944 raft_consensus.cc:3037] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:20.719455 19944 raft_consensus.cc:504] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.719537 19872 raft_consensus.cc:504] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.719977 19467 raft_consensus.cc:2379] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 68ab8c71129b4a3aaacbadf8447307a2 in current term 1: Already voted for candidate d92948900f73475b81fd739707311954 in this term.
I20260810 14:32:20.720057 19944 leader_election.cc:288] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [CANDIDATE]: Term 1 election: Requested vote from peers d92948900f73475b81fd739707311954 (127.6.46.194:37085), 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195:39551)
I20260810 14:32:20.720541 19240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae070e065114dc2bf52546416835166" candidate_uuid: "d92948900f73475b81fd739707311954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ab8c71129b4a3aaacbadf8447307a2"
I20260810 14:32:20.720826 19240 raft_consensus.cc:2379] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d92948900f73475b81fd739707311954 in current term 1: Already voted for candidate 68ab8c71129b4a3aaacbadf8447307a2 in this term.
I20260810 14:32:20.721074 19467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae070e065114dc2bf52546416835166" candidate_uuid: "68ab8c71129b4a3aaacbadf8447307a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92948900f73475b81fd739707311954"
I20260810 14:32:20.721336 19467 raft_consensus.cc:2379] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 68ab8c71129b4a3aaacbadf8447307a2 in current term 1: Already voted for candidate d92948900f73475b81fd739707311954 in this term.
I20260810 14:32:20.721307 19705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae070e065114dc2bf52546416835166" candidate_uuid: "68ab8c71129b4a3aaacbadf8447307a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6521c3bfbdbf4fc68aaad46b613e6020"
I20260810 14:32:20.721473 19705 raft_consensus.cc:3037] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:20.721975 19704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dae070e065114dc2bf52546416835166" candidate_uuid: "d92948900f73475b81fd739707311954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6521c3bfbdbf4fc68aaad46b613e6020"
I20260810 14:32:20.722744 19359 leader_election.cc:302] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d92948900f73475b81fd739707311954; no voters: 6521c3bfbdbf4fc68aaad46b613e6020, 68ab8c71129b4a3aaacbadf8447307a2
I20260810 14:32:20.723254 19872 leader_election.cc:288] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [CANDIDATE]: Term 1 election: Requested vote from peers 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193:35025), 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195:39551)
I20260810 14:32:20.723446 19997 raft_consensus.cc:2726] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 14:32:20.724108 19705 raft_consensus.cc:2454] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 68ab8c71129b4a3aaacbadf8447307a2 in term 1.
I20260810 14:32:20.724452 19135 leader_election.cc:302] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6521c3bfbdbf4fc68aaad46b613e6020, 68ab8c71129b4a3aaacbadf8447307a2; no voters: d92948900f73475b81fd739707311954
I20260810 14:32:20.724704 19944 raft_consensus.cc:2781] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:20.724853 19944 raft_consensus.cc:686] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 LEADER]: Becoming Leader. State: Replica: 68ab8c71129b4a3aaacbadf8447307a2, State: Running, Role: LEADER
I20260810 14:32:20.725116 19944 consensus_queue.cc:237] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.727746 18995 catalog_manager.cc:5495] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193). New cstate: current_term: 1 leader_uuid: "68ab8c71129b4a3aaacbadf8447307a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } health_report { overall_health: UNKNOWN } } }
I20260810 14:32:20.788268 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:20.779483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51748 (local address 127.6.46.193:35025)
0810 14:32:20.783913 (+ 4430us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:20.783919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:20.783940 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:32:20.784012 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:20.784017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:20.784086 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:20.784175 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:32:20.784531 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.785210 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.786683 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:20.787090 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:20.787200 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:20.787588 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:32:20.787710 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:32:20.787786 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:20.787924 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 14:32:20.788005 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4332,"thread_start_us":4242,"threads_started":1}
I20260810 14:32:20.791350 19705 raft_consensus.cc:1261] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Refusing update from remote peer 68ab8c71129b4a3aaacbadf8447307a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:20.791534 19467 raft_consensus.cc:1261] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Refusing update from remote peer 68ab8c71129b4a3aaacbadf8447307a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:20.792065 19922 consensus_queue.cc:1035] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:20.792572 19922 consensus_queue.cc:1035] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:20.810670 18992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:20.810904 18992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:20.814590 18992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1: Sending DeleteTablet for 3 replicas of tablet dae070e065114dc2bf52546416835166
I20260810 14:32:20.815322 19214 tablet_service.cc:1514] Processing DeleteTablet for tablet dae070e065114dc2bf52546416835166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:20 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:51728
I20260810 14:32:20.815582 19961 tablet_replica.cc:330] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2: stopping tablet replica
I20260810 14:32:20.815712 19667 tablet_service.cc:1514] Processing DeleteTablet for tablet dae070e065114dc2bf52546416835166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:20 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:49042
I20260810 14:32:20.815786 19961 raft_consensus.cc:2229] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:32:20.815968 19968 tablet_replica.cc:330] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020: stopping tablet replica
I20260810 14:32:20.816155 19968 raft_consensus.cc:2229] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:20.816244 19961 raft_consensus.cc:2258] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:20.816365 19968 raft_consensus.cc:2258] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:20.817124 19430 tablet_service.cc:1514] Processing DeleteTablet for tablet dae070e065114dc2bf52546416835166 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:20 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:60276
I20260810 14:32:20.817387 19966 tablet_replica.cc:330] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954: stopping tablet replica
I20260810 14:32:20.817569 19966 raft_consensus.cc:2229] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:20.817644 19968 ts_tablet_manager.cc:1905] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:20.817763 19966 raft_consensus.cc:2258] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:20.818732 19961 ts_tablet_manager.cc:1905] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:20.823590 19966 ts_tablet_manager.cc:1905] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:20.824016 19968 ts_tablet_manager.cc:1918] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:20.824164 19968 log.cc:1196] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/dae070e065114dc2bf52546416835166
I20260810 14:32:20.824559 19968 ts_tablet_manager.cc:1939] T dae070e065114dc2bf52546416835166 P 6521c3bfbdbf4fc68aaad46b613e6020: Deleting consensus metadata
I20260810 14:32:20.825709 18977 catalog_manager.cc:4841] TS 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195:39551): tablet dae070e065114dc2bf52546416835166 (table test-table [id=54a28a8660174e40a03a13b4617554fa]) successfully deleted
I20260810 14:32:20.827350 19961 ts_tablet_manager.cc:1918] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:20.827504 19961 log.cc:1196] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/dae070e065114dc2bf52546416835166
I20260810 14:32:20.827965 19961 ts_tablet_manager.cc:1939] T dae070e065114dc2bf52546416835166 P 68ab8c71129b4a3aaacbadf8447307a2: Deleting consensus metadata
I20260810 14:32:20.828598 19966 ts_tablet_manager.cc:1918] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:20.828727 19966 log.cc:1196] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/dae070e065114dc2bf52546416835166
I20260810 14:32:20.829113 18976 catalog_manager.cc:4841] TS 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193:35025): tablet dae070e065114dc2bf52546416835166 (table test-table [id=54a28a8660174e40a03a13b4617554fa]) successfully deleted
I20260810 14:32:20.829514 19966 ts_tablet_manager.cc:1939] T dae070e065114dc2bf52546416835166 P d92948900f73475b81fd739707311954: Deleting consensus metadata
I20260810 14:32:20.829546 18992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:32:20.830265 18992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:32:20.830433 18972 catalog_manager.cc:4841] TS d92948900f73475b81fd739707311954 (127.6.46.194:37085): tablet dae070e065114dc2bf52546416835166 (table test-table [id=54a28a8660174e40a03a13b4617554fa]) successfully deleted
I20260810 14:32:20.837319 19430 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3118a7c3754c4f8fbdc995eb2fe20d (DEFAULT_TABLE table=test-table [id=628c3c5442cc4104ab6cd1149cc3fbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:20.837432 19667 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3118a7c3754c4f8fbdc995eb2fe20d (DEFAULT_TABLE table=test-table [id=628c3c5442cc4104ab6cd1149cc3fbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:20.837785 19430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3118a7c3754c4f8fbdc995eb2fe20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:20.837811 19667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3118a7c3754c4f8fbdc995eb2fe20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:20.843871 19889 tablet_bootstrap.cc:492] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020: Bootstrap starting.
I20260810 14:32:20.843885 19214 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3118a7c3754c4f8fbdc995eb2fe20d (DEFAULT_TABLE table=test-table [id=628c3c5442cc4104ab6cd1149cc3fbd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:20.844334 19214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3118a7c3754c4f8fbdc995eb2fe20d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:20.845594 19889 tablet_bootstrap.cc:654] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:20.848120 19889 tablet_bootstrap.cc:492] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020: No bootstrap required, opened a new log
I20260810 14:32:20.848286 19889 ts_tablet_manager.cc:1397] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 14:32:20.849056 19889 raft_consensus.cc:348] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.849313 19889 raft_consensus.cc:374] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:20.849395 19889 raft_consensus.cc:729] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6521c3bfbdbf4fc68aaad46b613e6020, State: Initialized, Role: FOLLOWER
I20260810 14:32:20.849643 19889 consensus_queue.cc:260] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.849956 19888 tablet_bootstrap.cc:492] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2: Bootstrap starting.
I20260810 14:32:20.851074 19890 tablet_bootstrap.cc:492] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954: Bootstrap starting.
I20260810 14:32:20.851807 19888 tablet_bootstrap.cc:654] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:20.852643 19890 tablet_bootstrap.cc:654] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:20.853948 19889 ts_tablet_manager.cc:1428] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260810 14:32:20.854058 19888 tablet_bootstrap.cc:492] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2: No bootstrap required, opened a new log
I20260810 14:32:20.854213 19888 ts_tablet_manager.cc:1397] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20260810 14:32:20.855283 19888 raft_consensus.cc:348] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.855581 19888 raft_consensus.cc:374] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:20.855674 19888 raft_consensus.cc:729] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ab8c71129b4a3aaacbadf8447307a2, State: Initialized, Role: FOLLOWER
I20260810 14:32:20.855913 19888 consensus_queue.cc:260] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.856871 19888 ts_tablet_manager.cc:1428] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260810 14:32:20.863415 19890 tablet_bootstrap.cc:492] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954: No bootstrap required, opened a new log
I20260810 14:32:20.863592 19890 ts_tablet_manager.cc:1397] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20260810 14:32:20.864382 19890 raft_consensus.cc:348] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.864671 19890 raft_consensus.cc:374] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:20.864758 19890 raft_consensus.cc:729] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92948900f73475b81fd739707311954, State: Initialized, Role: FOLLOWER
I20260810 14:32:20.864996 19890 consensus_queue.cc:260] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.865548 19890 ts_tablet_manager.cc:1428] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 14:32:20.983773 19863 raft_consensus.cc:482] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:32:20.983965 19863 raft_consensus.cc:504] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.984711 19863 leader_election.cc:288] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193:35025), d92948900f73475b81fd739707311954 (127.6.46.194:37085)
I20260810 14:32:20.985198 19467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3118a7c3754c4f8fbdc995eb2fe20d" candidate_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92948900f73475b81fd739707311954" is_pre_election: true
I20260810 14:32:20.985224 19240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3118a7c3754c4f8fbdc995eb2fe20d" candidate_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ab8c71129b4a3aaacbadf8447307a2" is_pre_election: true
I20260810 14:32:20.985446 19467 raft_consensus.cc:2454] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6521c3bfbdbf4fc68aaad46b613e6020 in term 0.
I20260810 14:32:20.985455 19240 raft_consensus.cc:2454] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6521c3bfbdbf4fc68aaad46b613e6020 in term 0.
I20260810 14:32:20.985909 19583 leader_election.cc:302] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6521c3bfbdbf4fc68aaad46b613e6020, 68ab8c71129b4a3aaacbadf8447307a2; no voters:
I20260810 14:32:20.986222 19863 raft_consensus.cc:2781] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:32:20.986323 19863 raft_consensus.cc:482] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:32:20.986431 19863 raft_consensus.cc:3037] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:20.988374 19863 raft_consensus.cc:504] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.988893 19863 leader_election.cc:288] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [CANDIDATE]: Term 1 election: Requested vote from peers 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193:35025), d92948900f73475b81fd739707311954 (127.6.46.194:37085)
I20260810 14:32:20.989296 19240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3118a7c3754c4f8fbdc995eb2fe20d" candidate_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ab8c71129b4a3aaacbadf8447307a2"
I20260810 14:32:20.989324 19467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3118a7c3754c4f8fbdc995eb2fe20d" candidate_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92948900f73475b81fd739707311954"
I20260810 14:32:20.989506 19240 raft_consensus.cc:3037] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:20.989557 19467 raft_consensus.cc:3037] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:20.991896 19240 raft_consensus.cc:2454] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6521c3bfbdbf4fc68aaad46b613e6020 in term 1.
I20260810 14:32:20.992233 19583 leader_election.cc:302] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6521c3bfbdbf4fc68aaad46b613e6020, 68ab8c71129b4a3aaacbadf8447307a2; no voters:
I20260810 14:32:20.992472 19863 raft_consensus.cc:2781] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:20.992515 19467 raft_consensus.cc:2454] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6521c3bfbdbf4fc68aaad46b613e6020 in term 1.
I20260810 14:32:20.992905 19863 raft_consensus.cc:686] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 LEADER]: Becoming Leader. State: Replica: 6521c3bfbdbf4fc68aaad46b613e6020, State: Running, Role: LEADER
I20260810 14:32:20.993175 19863 consensus_queue.cc:237] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } }
I20260810 14:32:20.995504 18992 catalog_manager.cc:5495] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195). New cstate: current_term: 1 leader_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6521c3bfbdbf4fc68aaad46b613e6020" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 39551 } health_report { overall_health: HEALTHY } } }
I20260810 14:32:21.047482 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:21.038662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.6.46.195:39551)
0810 14:32:21.042456 (+ 3794us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:21.042460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:21.042475 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:32:21.042524 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:21.042527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:21.042585 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:21.042664 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:32:21.043093 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:21.043950 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:21.045788 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:21.046257 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:21.046350 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:21.046875 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:32:21.046979 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:32:21.047041 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:21.047169 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 14:32:21.047228 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3694,"thread_start_us":3561,"threads_started":1}
I20260810 14:32:21.056402 19240 raft_consensus.cc:1261] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Refusing update from remote peer 6521c3bfbdbf4fc68aaad46b613e6020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:21.056628 19471 raft_consensus.cc:1261] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Refusing update from remote peer 6521c3bfbdbf4fc68aaad46b613e6020: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:21.057054 20014 consensus_queue.cc:1035] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ab8c71129b4a3aaacbadf8447307a2" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:21.057430 19863 consensus_queue.cc:1035] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92948900f73475b81fd739707311954" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:21.140865 18991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:21.141064 18991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:21.143936 18991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e51fef1490e4a1f9644678f1d564ea1: Sending DeleteTablet for 3 replicas of tablet 7d3118a7c3754c4f8fbdc995eb2fe20d
I20260810 14:32:21.144569 19214 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3118a7c3754c4f8fbdc995eb2fe20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:21 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:51728
I20260810 14:32:21.144954 19667 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3118a7c3754c4f8fbdc995eb2fe20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:21 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:49042
I20260810 14:32:21.144963 19430 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3118a7c3754c4f8fbdc995eb2fe20d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:21 UTC) from {username='oryx', principal='oryx/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:60276
I20260810 14:32:21.145009 19961 tablet_replica.cc:330] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2: stopping tablet replica
I20260810 14:32:21.145221 19966 tablet_replica.cc:330] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954: stopping tablet replica
I20260810 14:32:21.145215 19961 raft_consensus.cc:2229] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:21.145342 19966 raft_consensus.cc:2229] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:21.145432 19961 raft_consensus.cc:2258] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:21.145479 19966 raft_consensus.cc:2258] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:21.145738 19968 tablet_replica.cc:330] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020: stopping tablet replica
I20260810 14:32:21.145905 19968 raft_consensus.cc:2229] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:32:21.146270 19968 raft_consensus.cc:2258] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:21.146325 19966 ts_tablet_manager.cc:1905] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:21.148073 19961 ts_tablet_manager.cc:1905] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:21.150907 19968 ts_tablet_manager.cc:1905] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:21.151295 19966 ts_tablet_manager.cc:1918] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:32:21.151427 19966 log.cc:1196] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/7d3118a7c3754c4f8fbdc995eb2fe20d
I20260810 14:32:21.151850 19966 ts_tablet_manager.cc:1939] T 7d3118a7c3754c4f8fbdc995eb2fe20d P d92948900f73475b81fd739707311954: Deleting consensus metadata
I20260810 14:32:21.152806 18972 catalog_manager.cc:4841] TS d92948900f73475b81fd739707311954 (127.6.46.194:37085): tablet 7d3118a7c3754c4f8fbdc995eb2fe20d (table test-table [id=628c3c5442cc4104ab6cd1149cc3fbd5]) successfully deleted
I20260810 14:32:21.154896 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 19105
I20260810 14:32:21.156667 19968 ts_tablet_manager.cc:1918] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:32:21.156800 19968 log.cc:1196] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/7d3118a7c3754c4f8fbdc995eb2fe20d
I20260810 14:32:21.157217 19968 ts_tablet_manager.cc:1939] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 6521c3bfbdbf4fc68aaad46b613e6020: Deleting consensus metadata
I20260810 14:32:21.157655 19961 ts_tablet_manager.cc:1918] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 14:32:21.157840 19961 log.cc:1196] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/7d3118a7c3754c4f8fbdc995eb2fe20d
I20260810 14:32:21.158250 18977 catalog_manager.cc:4841] TS 6521c3bfbdbf4fc68aaad46b613e6020 (127.6.46.195:39551): tablet 7d3118a7c3754c4f8fbdc995eb2fe20d (table test-table [id=628c3c5442cc4104ab6cd1149cc3fbd5]) successfully deleted
I20260810 14:32:21.158728 19961 ts_tablet_manager.cc:1939] T 7d3118a7c3754c4f8fbdc995eb2fe20d P 68ab8c71129b4a3aaacbadf8447307a2: Deleting consensus metadata
I20260810 14:32:21.159843 18976 catalog_manager.cc:4841] TS 68ab8c71129b4a3aaacbadf8447307a2 (127.6.46.193:35025): tablet 7d3118a7c3754c4f8fbdc995eb2fe20d (table test-table [id=628c3c5442cc4104ab6cd1149cc3fbd5]) successfully deleted
Aug 10 14:32:21 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.46.193@KRBTEST.COM
I20260810 14:32:21.172397 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:21.156037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51762 (local address 127.6.46.193:35025)
0810 14:32:21.156202 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:21.156209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:21.158667 (+ 2458us) server_negotiation.cc:407] Connection header received
0810 14:32:21.158835 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:21.158839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:21.158904 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:21.158998 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:21.160267 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:21.160999 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:21.162169 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:21.162605 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:21.162715 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:21.166811 (+ 4096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:21.166823 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:21.166837 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:21.166868 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:21.169797 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:21.170457 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:21.170465 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:21.170467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:21.170589 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:21.170942 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:21.170945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:21.170947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:21.171298 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:21.171621 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:21.171853 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 14:32:21.172091 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:32:21.173105 19310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:21.324883 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 19105
I20260810 14:32:21.356840 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 19326
Aug 10 14:32:21 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.46.194@KRBTEST.COM
I20260810 14:32:21.371325 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:21.357672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.6.46.194:37085)
0810 14:32:21.358190 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:21.358195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:21.358214 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:32:21.358274 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:21.358278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:21.358344 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:21.358455 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:21.359595 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:21.360360 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:21.361495 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:21.361909 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:21.362171 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:21.365915 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:21.365936 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:21.365957 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:21.366060 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:21.368982 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:21.369576 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:21.369583 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:21.369585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:21.369750 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:21.370057 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:21.370062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:21.370064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:21.370467 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:21.370701 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:21.370876 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 14:32:21.371038 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":243,"threads_started":1}
I20260810 14:32:21.371953 19545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:21.489339 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 19326
I20260810 14:32:21.517895 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 19557
Aug 10 14:32:21 dist-test-slave-q9h9 krb5kdc[18917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.46.195@KRBTEST.COM
I20260810 14:32:21.532688 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:21.518671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49060 (local address 127.6.46.195:39551)
0810 14:32:21.518845 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:21.518850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:21.518983 (+ 133us) server_negotiation.cc:407] Connection header received
0810 14:32:21.519119 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:21.519123 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:21.519203 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:21.519305 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:21.520639 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:21.521439 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:21.522743 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:21.523241 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:21.523356 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:21.527419 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:21.527434 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:21.527452 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:21.527498 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:21.530384 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:21.530983 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:21.530990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:21.530992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:21.531149 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:21.531434 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:21.531438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:21.531440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:21.531794 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:21.532028 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:21.532215 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 14:32:21.532372 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:32:21.533294 19767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:21.657306 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 19557
I20260810 14:32:21.684065 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 18944
I20260810 14:32:21.685235 19064 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:21.798606 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 18944
2026-08-10T14:32:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6173 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:32:21 dist-test-slave-q9h9 krb5kdc[20041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:32:21 dist-test-slave-q9h9 krb5kdc[20041](info): set up 2 sockets
Aug 10 14:32:21 dist-test-slave-q9h9 krb5kdc[20041](info): commencing operation
krb5kdc: starting...
W20260810 14:32:23.035717 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:32:23 dist-test-slave-q9h9 krb5kdc[20041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:32:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:32:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:32:23.253943 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:34881
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:38227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:34881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:23.569556 20061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:23.569823 20061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:23.569947 20061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:23.576944 20061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:32:23.577039 20061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:23.577093 20061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:23.577170 20061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:32:23.577232 20061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:32:23.586678 20061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:38227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:34881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:34881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:23.587955 20061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:23.589262 20061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:23.600528 20068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:23.600632 20067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:23.601006 20070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:23.601588 20061 server_base.cc:1034] running on GCE node
I20260810 14:32:23.602335 20061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:23.603588 20061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:23.604803 20061 hybrid_clock.cc:648] HybridClock initialized: now 1786372343604753 us; error 36 us; skew 500 ppm
I20260810 14:32:23.605114 20061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T14:32:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1874 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:32:23 dist-test-slave-q9h9 krb5kdc[20080](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:32:23 dist-test-slave-q9h9 krb5kdc[20080](info): set up 2 sockets
Aug 10 14:32:23 dist-test-slave-q9h9 krb5kdc[20080](info): commencing operation
krb5kdc: starting...
W20260810 14:32:24.916595 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:32:25 dist-test-slave-q9h9 krb5kdc[20080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:32:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:32:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:32:25.143855 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:35109
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:35109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:25.461359 20113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:25.461643 20113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:25.461769 20113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:25.468717 20113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:32:25.468828 20113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:25.468909 20113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:25.468983 20113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:32:25.469054 20113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:32:25.478487 20113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:35109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:35109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:25.479708 20113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:25.480964 20113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:25.492520 20118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:25.492547 20119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:25.492604 20121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:25.493265 20113 server_base.cc:1034] running on GCE node
I20260810 14:32:25.494148 20113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:25.495543 20113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:25.496762 20113 hybrid_clock.cc:648] HybridClock initialized: now 1786372345496715 us; error 38 us; skew 500 ppm
I20260810 14:32:25.497097 20113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:25 dist-test-slave-q9h9 krb5kdc[20080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372345, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:25.504410 20113 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:32:25.505870 20113 webserver.cc:460] Webserver started at http://127.6.46.254:34643/ using document root <none> and password file <none>
I20260810 14:32:25.506578 20113 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:25.506692 20113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:25.506996 20113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:25.509881 20113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "400d50550f51497499abb7299dac09ea"
format_stamp: "Formatted at 2026-08-10 14:32:25 on dist-test-slave-q9h9"
server_key: "e1c0a2089110b45668506204a5b91ac7"
server_key_iv: "5a8f8b4a50229056416068248e113177"
server_key_version: "encryptionkey@0"
I20260810 14:32:25.510699 20113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "400d50550f51497499abb7299dac09ea"
format_stamp: "Formatted at 2026-08-10 14:32:25 on dist-test-slave-q9h9"
server_key: "e1c0a2089110b45668506204a5b91ac7"
server_key_iv: "5a8f8b4a50229056416068248e113177"
server_key_version: "encryptionkey@0"
I20260810 14:32:25.515722 20113 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 14:32:25.519016 20128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:25.520226 20113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:32:25.520376 20113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "400d50550f51497499abb7299dac09ea"
format_stamp: "Formatted at 2026-08-10 14:32:25 on dist-test-slave-q9h9"
server_key: "e1c0a2089110b45668506204a5b91ac7"
server_key_iv: "5a8f8b4a50229056416068248e113177"
server_key_version: "encryptionkey@0"
I20260810 14:32:25.520546 20113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:25.539707 20113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:25.543432 20113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:25.543676 20113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:25.561043 20113 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:35109
I20260810 14:32:25.561072 20207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:35109 every 8 connection(s)
I20260810 14:32:25.562440 20113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:32:25.565213 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 20113
I20260810 14:32:25.565430 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:32:25.566123 6331 external_mini_cluster.cc:1428] Setting key cbea8822bb3a9e7c427a482e8f9330ed
I20260810 14:32:25.566300 20209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:25.580157 20209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea: Bootstrap starting.
Aug 10 14:32:25 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:32:25.584015 20209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:25.585343 20209 log.cc:824] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:25.588038 20209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea: No bootstrap required, opened a new log
I20260810 14:32:25.589239 20212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:25.568005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.6.46.254:35109)
0810 14:32:25.568760 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:25.568775 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:25.568831 (+ 56us) server_negotiation.cc:407] Connection header received
0810 14:32:25.571094 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:25.571121 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:25.571612 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:25.572034 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:25.573351 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:25.575320 (+ 1969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:25.576539 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:25.577584 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:25.577771 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:25.581674 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:25.581688 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:25.581708 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:25.581764 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:25.585470 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:25.586126 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:25.586135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:25.586143 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:25.586321 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:25.586687 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:25.586690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:25.586692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:25.587186 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:25.587441 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:25.587890 (+ 449us) server_negotiation.cc:299] Negotiation successful
0810 14:32:25.588321 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":312,"threads_started":1}
I20260810 14:32:25.595664 20209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400d50550f51497499abb7299dac09ea" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35109 } }
I20260810 14:32:25.595996 20209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:25.596093 20209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 400d50550f51497499abb7299dac09ea, State: Initialized, Role: FOLLOWER
I20260810 14:32:25.596585 20209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400d50550f51497499abb7299dac09ea" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35109 } }
I20260810 14:32:25.596809 20209 raft_consensus.cc:388] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:32:25.596931 20209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:32:25.597079 20209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:25.599180 20209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400d50550f51497499abb7299dac09ea" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35109 } }
I20260810 14:32:25.599627 20209 leader_election.cc:302] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 400d50550f51497499abb7299dac09ea; no voters:
I20260810 14:32:25.600152 20209 leader_election.cc:288] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:32:25.600260 20218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:25.600869 20218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 1 LEADER]: Becoming Leader. State: Replica: 400d50550f51497499abb7299dac09ea, State: Running, Role: LEADER
I20260810 14:32:25.601358 20218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400d50550f51497499abb7299dac09ea" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35109 } }
I20260810 14:32:25.601992 20209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:32:25.603818 20219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "400d50550f51497499abb7299dac09ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400d50550f51497499abb7299dac09ea" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35109 } } }
I20260810 14:32:25.603864 20222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 400d50550f51497499abb7299dac09ea. Latest consensus state: current_term: 1 leader_uuid: "400d50550f51497499abb7299dac09ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400d50550f51497499abb7299dac09ea" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35109 } } }
I20260810 14:32:25.604259 20222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:25.604259 20219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:25.606407 20229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:32:25.611656 20229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:32:25.619160 20229 catalog_manager.cc:1347] Generated new cluster ID: a822f8edac4741ed8dd4373c52ce6712
I20260810 14:32:25.619258 20229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:32:25.631886 20229 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:32:25.633816 20229 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:32:25.647125 20229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea: Generated new TSK 0
I20260810 14:32:25.648263 20229 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:32:25.712700 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:35109
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:26.030109 20251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:26.030355 20251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:26.030474 20251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:26.037281 20251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:26.037374 20251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:26.037519 20251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:32:26.046486 20251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:35109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:26.047602 20251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:26.048796 20251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:26.060604 20264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:26.060686 20258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:26.060633 20261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:26.061688 20251 server_base.cc:1034] running on GCE node
I20260810 14:32:26.062250 20251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:26.063051 20251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:26.064273 20251 hybrid_clock.cc:648] HybridClock initialized: now 1786372346064234 us; error 27 us; skew 500 ppm
I20260810 14:32:26.064590 20251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:26 dist-test-slave-q9h9 krb5kdc[20080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372346, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:26.067607 20251 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:32:26.068903 20251 webserver.cc:460] Webserver started at http://127.6.46.193:36539/ using document root <none> and password file <none>
I20260810 14:32:26.069509 20251 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:26.069634 20251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:26.069921 20251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:26.072634 20251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "256be682e8d741d5b0ee963fdd26d9bb"
format_stamp: "Formatted at 2026-08-10 14:32:26 on dist-test-slave-q9h9"
server_key: "121de39b234b0458eecb0f788a3d79a5"
server_key_iv: "545a294a04e38b163b4ee020e23482f0"
server_key_version: "encryptionkey@0"
I20260810 14:32:26.073361 20251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "256be682e8d741d5b0ee963fdd26d9bb"
format_stamp: "Formatted at 2026-08-10 14:32:26 on dist-test-slave-q9h9"
server_key: "121de39b234b0458eecb0f788a3d79a5"
server_key_iv: "545a294a04e38b163b4ee020e23482f0"
server_key_version: "encryptionkey@0"
I20260810 14:32:26.078122 20251 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 14:32:26.081277 20275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:26.082527 20251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:32:26.082683 20251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "256be682e8d741d5b0ee963fdd26d9bb"
format_stamp: "Formatted at 2026-08-10 14:32:26 on dist-test-slave-q9h9"
server_key: "121de39b234b0458eecb0f788a3d79a5"
server_key_iv: "545a294a04e38b163b4ee020e23482f0"
server_key_version: "encryptionkey@0"
I20260810 14:32:26.082856 20251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:26.105871 20251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:26.109648 20251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:26.109905 20251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:26.110833 20251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:26.112258 20251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:26.112344 20251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:26.112464 20251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:26.112519 20251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:26.142942 20251 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:34139
I20260810 14:32:26.143013 20459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:34139 every 8 connection(s)
I20260810 14:32:26.144217 20251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:32:26.146185 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 20251
I20260810 14:32:26.146414 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:32:26.147193 6331 external_mini_cluster.cc:1428] Setting key 3837c9b109612e72c4e12552a017538f
Aug 10 14:32:26 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372346, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:32:26.164901 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:26.146804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:42021 (local address 127.6.46.254:35109)
0810 14:32:26.147287 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:26.147297 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:26.148348 (+ 1051us) server_negotiation.cc:407] Connection header received
0810 14:32:26.150637 (+ 2289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:26.150641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:26.150711 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:26.150795 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:26.152638 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:26.153323 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:26.155331 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:26.155726 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:26.155821 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:26.159564 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:26.159569 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:26.159572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:26.159604 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:26.162054 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:26.162738 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:26.162743 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:26.162745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:26.162859 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:26.163270 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:26.163274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:26.163276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:26.163535 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:26.163668 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:26.164345 (+ 677us) server_negotiation.cc:299] Negotiation successful
0810 14:32:26.164558 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":223,"threads_started":1}
I20260810 14:32:26.165835 20467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:26.147406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:42021)
0810 14:32:26.148178 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:26.148235 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:26.150347 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:26.151057 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:26.151068 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:26.151757 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:26.152486 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:26.152508 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:26.153465 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:26.153470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:26.155189 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:26.155202 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:26.155936 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:26.155942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:26.156187 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:26.156844 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:26.156872 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:26.159350 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:26.162183 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:26.162192 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:26.162208 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:26.162599 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:26.162967 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:26.162972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:26.162975 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:26.163149 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:26.163682 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:26.163690 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:26.163982 (+ 292us) client_negotiation.cc:769] Sending connection context
0810 14:32:26.164206 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 14:32:26.164702 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":314,"threads_started":1}
I20260810 14:32:26.167438 20462 heartbeater.cc:347] Connected to a master server at 127.6.46.254:35109
I20260810 14:32:26.167723 20462 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:26.168372 20462 heartbeater.cc:511] Master 127.6.46.254:35109 requested a full tablet report, sending...
I20260810 14:32:26.170717 20152 ts_manager.cc:194] Registered new tserver with Master: 256be682e8d741d5b0ee963fdd26d9bb (127.6.46.193:34139)
I20260810 14:32:26.172712 20152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:42021
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:32:26.202919 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:35109
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:26.515961 20472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:26.516213 20472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:26.516346 20472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:26.523242 20472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:26.523335 20472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:26.523478 20472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:32:26.532631 20472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:35109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:26.533730 20472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:26.534996 20472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:26.547362 20481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:26.547338 20482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:26.547469 20486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:26.548281 20472 server_base.cc:1034] running on GCE node
I20260810 14:32:26.548828 20472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:26.549672 20472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:26.550872 20472 hybrid_clock.cc:648] HybridClock initialized: now 1786372346550829 us; error 34 us; skew 500 ppm
I20260810 14:32:26.551179 20472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:26 dist-test-slave-q9h9 krb5kdc[20080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372346, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:26.554103 20472 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:32:26.555423 20472 webserver.cc:460] Webserver started at http://127.6.46.194:32827/ using document root <none> and password file <none>
I20260810 14:32:26.556028 20472 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:26.556151 20472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:26.556434 20472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:26.559132 20472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "d769cfbeff2541069a44b6882100d263"
format_stamp: "Formatted at 2026-08-10 14:32:26 on dist-test-slave-q9h9"
server_key: "43cb816545e5ad3a385a021682dd7c31"
server_key_iv: "713d6b5987670e65abd58f2dd1672893"
server_key_version: "encryptionkey@0"
I20260810 14:32:26.559861 20472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "d769cfbeff2541069a44b6882100d263"
format_stamp: "Formatted at 2026-08-10 14:32:26 on dist-test-slave-q9h9"
server_key: "43cb816545e5ad3a385a021682dd7c31"
server_key_iv: "713d6b5987670e65abd58f2dd1672893"
server_key_version: "encryptionkey@0"
I20260810 14:32:26.564813 20472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 14:32:26.568120 20497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:26.569386 20472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:32:26.569531 20472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "d769cfbeff2541069a44b6882100d263"
format_stamp: "Formatted at 2026-08-10 14:32:26 on dist-test-slave-q9h9"
server_key: "43cb816545e5ad3a385a021682dd7c31"
server_key_iv: "713d6b5987670e65abd58f2dd1672893"
server_key_version: "encryptionkey@0"
I20260810 14:32:26.569700 20472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:26.585422 20472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:26.589183 20472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:26.589432 20472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:26.590337 20472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:26.591578 20472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:26.591657 20472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:26.591786 20472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:26.591840 20472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:26.621575 20472 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:45229
I20260810 14:32:26.621634 20692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:45229 every 8 connection(s)
I20260810 14:32:26.622820 20472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:32:26.632951 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 20472
I20260810 14:32:26.633164 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:32:26.633596 6331 external_mini_cluster.cc:1428] Setting key 69e1ab4f6fcf87101270283ca8f7561b
Aug 10 14:32:26 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372346, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:32:26.642120 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:26.625289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:44315 (local address 127.6.46.254:35109)
0810 14:32:26.625501 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:26.625505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:26.626605 (+ 1100us) server_negotiation.cc:407] Connection header received
0810 14:32:26.628389 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:26.628396 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:26.628477 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:26.628569 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:26.630096 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:26.630853 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:26.632427 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:26.632860 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:26.632974 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:26.636700 (+ 3726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:26.636705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:26.636709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:26.636741 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:26.639223 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:26.639917 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:26.639921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:26.639923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:26.640068 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:26.640489 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:26.640493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:26.640495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:26.640740 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:26.640879 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:26.641536 (+ 657us) server_negotiation.cc:299] Negotiation successful
0810 14:32:26.641756 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:32:26.643056 20695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:26.625728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:44315)
0810 14:32:26.626448 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:26.626492 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:26.628120 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:26.628834 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:26.628842 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:26.629341 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:26.629898 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:26.629916 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:26.631026 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:26.631030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:26.632268 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:26.632281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:26.633092 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:26.633099 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:26.633355 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:26.634057 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:26.634086 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:26.636466 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:26.639360 (+ 2894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:26.639369 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:26.639385 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:26.639796 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:26.640184 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:26.640189 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:26.640193 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:26.640375 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:26.640872 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:26.640880 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:26.641194 (+ 314us) client_negotiation.cc:769] Sending connection context
0810 14:32:26.641438 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 14:32:26.641932 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":324,"threads_started":1}
I20260810 14:32:26.644479 20693 heartbeater.cc:347] Connected to a master server at 127.6.46.254:35109
I20260810 14:32:26.644757 20693 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:26.645426 20693 heartbeater.cc:511] Master 127.6.46.254:35109 requested a full tablet report, sending...
I20260810 14:32:26.646907 20152 ts_manager.cc:194] Registered new tserver with Master: d769cfbeff2541069a44b6882100d263 (127.6.46.194:45229)
I20260810 14:32:26.647715 20152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:44315
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:32:26.685297 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:35109
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:27.001555 20700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:27.001802 20700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:27.001928 20700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:27.008924 20700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:27.009023 20700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:27.009138 20700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:32:27.018208 20700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:35109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:27.019299 20700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:27.020489 20700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:27.032956 20708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:27.033001 20706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:27.033206 20712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:27.033887 20700 server_base.cc:1034] running on GCE node
I20260810 14:32:27.034437 20700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:27.035321 20700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:27.036527 20700 hybrid_clock.cc:648] HybridClock initialized: now 1786372347036480 us; error 39 us; skew 500 ppm
I20260810 14:32:27.036813 20700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:27 dist-test-slave-q9h9 krb5kdc[20080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372347, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:27.039709 20700 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:32:27.041057 20700 webserver.cc:460] Webserver started at http://127.6.46.195:44737/ using document root <none> and password file <none>
I20260810 14:32:27.041669 20700 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:27.041785 20700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:27.042119 20700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:27.044829 20700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "9264f29486354e688339282e2f7fc67e"
format_stamp: "Formatted at 2026-08-10 14:32:27 on dist-test-slave-q9h9"
server_key: "aacf8377a284cdb430b91817c9fd6546"
server_key_iv: "655963f3aae406df4a53c6a837898a26"
server_key_version: "encryptionkey@0"
I20260810 14:32:27.045594 20700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "9264f29486354e688339282e2f7fc67e"
format_stamp: "Formatted at 2026-08-10 14:32:27 on dist-test-slave-q9h9"
server_key: "aacf8377a284cdb430b91817c9fd6546"
server_key_iv: "655963f3aae406df4a53c6a837898a26"
server_key_version: "encryptionkey@0"
I20260810 14:32:27.050316 20700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260810 14:32:27.053501 20723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:27.054778 20700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:32:27.054929 20700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "9264f29486354e688339282e2f7fc67e"
format_stamp: "Formatted at 2026-08-10 14:32:27 on dist-test-slave-q9h9"
server_key: "aacf8377a284cdb430b91817c9fd6546"
server_key_iv: "655963f3aae406df4a53c6a837898a26"
server_key_version: "encryptionkey@0"
I20260810 14:32:27.055107 20700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:27.089584 20700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:27.093266 20700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:27.093531 20700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:27.094483 20700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:27.095866 20700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:27.095963 20700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:27.096086 20700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:27.096134 20700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:27.126916 20700 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:45057
I20260810 14:32:27.126967 20911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:45057 every 8 connection(s)
I20260810 14:32:27.128162 20700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:32:27.136799 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 20700
I20260810 14:32:27.136965 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:32:27.137380 6331 external_mini_cluster.cc:1428] Setting key 80e5a95d88aee79e1a93323de3d74f6c
Aug 10 14:32:27 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372347, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:32:27.148689 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:27.130568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:35133 (local address 127.6.46.254:35109)
0810 14:32:27.130778 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:27.130784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:27.131979 (+ 1195us) server_negotiation.cc:407] Connection header received
0810 14:32:27.133735 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:27.133740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:27.133816 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:27.133921 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:27.135442 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:27.136192 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:27.137932 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:27.138510 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:27.138635 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:27.142277 (+ 3642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:27.142283 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:27.142287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:27.142327 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:27.145604 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:27.146454 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:27.146460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:27.146464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:27.146609 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:27.147005 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:27.147009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:27.147012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:27.147385 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:27.147533 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:27.148251 (+ 718us) server_negotiation.cc:299] Negotiation successful
0810 14:32:27.148446 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 14:32:27.149524 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:27.131115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:35133)
0810 14:32:27.131815 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:27.131853 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:27.133453 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:27.134178 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:27.134187 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:27.134680 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:27.135253 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:27.135271 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:27.136358 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:27.136363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:27.137765 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:27.137782 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:27.138751 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:27.138759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:27.139070 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:27.139769 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:27.139796 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:27.142069 (+ 2273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:27.145767 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:27.145778 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:27.145789 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:27.146332 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:27.146708 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:27.146711 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:27.146713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:27.146893 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:27.147538 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:27.147545 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:27.147857 (+ 312us) client_negotiation.cc:769] Sending connection context
0810 14:32:27.148095 (+ 238us) client_negotiation.cc:240] Negotiation successful
0810 14:32:27.148464 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":277,"threads_started":1}
I20260810 14:32:27.150889 20912 heartbeater.cc:347] Connected to a master server at 127.6.46.254:35109
I20260810 14:32:27.151165 20912 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:27.151854 20912 heartbeater.cc:511] Master 127.6.46.254:35109 requested a full tablet report, sending...
I20260810 14:32:27.153174 20152 ts_manager.cc:194] Registered new tserver with Master: 9264f29486354e688339282e2f7fc67e (127.6.46.195:45057)
I20260810 14:32:27.153960 20152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:35133
I20260810 14:32:27.164674 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:32:27.175766 20462 heartbeater.cc:503] Master 127.6.46.254:35109 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab.
Entry for principal oryx/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab.
WARNING: no policy specified for oryx/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab.
Entry for principal oryx/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab.
WARNING: no policy specified for oryx/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab.
Entry for principal oryx/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab.
I20260810 14:32:27.300503 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 20251
Aug 10 14:32:27 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:32:27.316119 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:27.301615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35874 (local address 127.6.46.193:34139)
0810 14:32:27.302406 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:27.302413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:27.302439 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:32:27.302509 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:27.302516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:27.302754 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:27.302924 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:27.304354 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:27.305343 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:27.306690 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:27.307307 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:27.307423 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:27.311290 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:27.311300 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:27.311314 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:27.311345 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:27.313837 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:27.314469 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:27.314475 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:27.314478 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:27.314608 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:27.314903 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:27.314906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:27.314908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:27.315237 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:27.315454 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:27.315647 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 14:32:27.315839 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":382,"threads_started":1}
I20260810 14:32:27.317373 20454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:27.397925 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 20251
I20260810 14:32:27.423213 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 20472
Aug 10 14:32:27 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:32:27.438196 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:27.424087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42034 (local address 127.6.46.194:45229)
0810 14:32:27.424616 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:27.424623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:27.424659 (+ 36us) server_negotiation.cc:407] Connection header received
0810 14:32:27.424719 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:27.424726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:27.424927 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:27.425099 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:27.426370 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:27.427365 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:27.428618 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:27.429148 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:27.429250 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:27.433120 (+ 3870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:27.433132 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:27.433147 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:27.433181 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:27.435903 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:27.436471 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:27.436478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:27.436480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:27.436592 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:27.436890 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:27.436893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:27.436895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:27.437203 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:27.437434 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:27.437689 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 14:32:27.437874 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":253,"threads_started":1}
I20260810 14:32:27.439335 20683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:27.518935 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 20472
I20260810 14:32:27.536872 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 20700
Aug 10 14:32:27 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:32:27.552138 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:27.537880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52404 (local address 127.6.46.195:45057)
0810 14:32:27.538589 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:27.538595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:27.538626 (+ 31us) server_negotiation.cc:407] Connection header received
0810 14:32:27.538691 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:27.538697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:27.538896 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:27.539029 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:27.540245 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:27.541280 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:27.542538 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:27.543095 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:27.543204 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:27.547260 (+ 4056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:27.547271 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:27.547285 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:27.547322 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:27.549829 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:27.550461 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:27.550468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:27.550470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:27.550604 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:27.550889 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:27.550892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:27.550894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:27.551230 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:27.551472 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:27.551659 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 14:32:27.551853 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":397,"threads_started":1}
I20260810 14:32:27.553373 20905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:27.636030 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 20700
I20260810 14:32:27.657100 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 20113
I20260810 14:32:27.658248 20201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:27.756290 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 20113
I20260810 14:32:27.779768 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:35109
--webserver_interface=127.6.46.254
--webserver_port=34643
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:35109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:28.096565 20937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:28.096822 20937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:28.096938 20937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:28.104022 20937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:32:28.104121 20937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:28.104174 20937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:28.104236 20937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:32:28.104279 20937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:32:28.113344 20937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:35109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:35109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=34643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:28.114413 20937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:28.115607 20937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:28.126513 20948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:28.126629 20949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:28.127010 20953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:28.127439 20937 server_base.cc:1034] running on GCE node
I20260810 14:32:28.128172 20937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:28.129467 20937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:28.130673 20937 hybrid_clock.cc:648] HybridClock initialized: now 1786372348130626 us; error 29 us; skew 500 ppm
I20260810 14:32:28.130991 20937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:28 dist-test-slave-q9h9 krb5kdc[20080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372348, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:28.133931 20937 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:32:28.135298 20937 webserver.cc:460] Webserver started at http://127.6.46.254:34643/ using document root <none> and password file <none>
I20260810 14:32:28.135928 20937 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:28.136055 20937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:28.141090 20937 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 14:32:28.143759 20968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:28.144941 20937 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:32:28.145097 20937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "400d50550f51497499abb7299dac09ea"
format_stamp: "Formatted at 2026-08-10 14:32:25 on dist-test-slave-q9h9"
server_key: "e1c0a2089110b45668506204a5b91ac7"
server_key_iv: "5a8f8b4a50229056416068248e113177"
server_key_version: "encryptionkey@0"
I20260810 14:32:28.145995 20937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:28.163532 20937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:28.167339 20937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:28.167573 20937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:28.184775 20937 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:35109
I20260810 14:32:28.184798 21054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:35109 every 8 connection(s)
I20260810 14:32:28.186306 20937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:32:28.192809 21055 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:32:28.193471 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 20937
I20260810 14:32:28.194260 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:34139
--local_ip_for_outbound_sockets=127.6.46.193
--tserver_master_addrs=127.6.46.254:35109
--webserver_port=36539
--webserver_interface=127.6.46.193
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.46.193
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:32:28.196074 21055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea: Bootstrap starting.
I20260810 14:32:28.216953 21055 log.cc:824] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:28.226768 21055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:32:28.227325 21055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea: Bootstrap complete.
I20260810 14:32:28.235989 21055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400d50550f51497499abb7299dac09ea" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35109 } }
I20260810 14:32:28.236773 21055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 400d50550f51497499abb7299dac09ea, State: Initialized, Role: FOLLOWER
I20260810 14:32:28.237407 21055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400d50550f51497499abb7299dac09ea" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35109 } }
I20260810 14:32:28.237622 21055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:32:28.237747 21055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:32:28.237908 21055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:32:28.241437 21055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400d50550f51497499abb7299dac09ea" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35109 } }
I20260810 14:32:28.241900 21055 leader_election.cc:302] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 400d50550f51497499abb7299dac09ea; no voters:
I20260810 14:32:28.242466 21055 leader_election.cc:288] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 14:32:28.242647 21060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:32:28.243685 21060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [term 2 LEADER]: Becoming Leader. State: Replica: 400d50550f51497499abb7299dac09ea, State: Running, Role: LEADER
I20260810 14:32:28.244326 21055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:32:28.244277 21060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400d50550f51497499abb7299dac09ea" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35109 } }
I20260810 14:32:28.246970 21063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 400d50550f51497499abb7299dac09ea. Latest consensus state: current_term: 2 leader_uuid: "400d50550f51497499abb7299dac09ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400d50550f51497499abb7299dac09ea" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35109 } } }
I20260810 14:32:28.247421 21063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:28.247347 21061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "400d50550f51497499abb7299dac09ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "400d50550f51497499abb7299dac09ea" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35109 } } }
I20260810 14:32:28.247701 21061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:28.257987 21070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:32:28.263475 21070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:32:28.264829 21070 catalog_manager.cc:1259] Loaded cluster ID: a822f8edac4741ed8dd4373c52ce6712
I20260810 14:32:28.264976 21070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:32:28.268905 21070 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:32:28.270361 21070 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 400d50550f51497499abb7299dac09ea: Loaded TSK: 0
I20260810 14:32:28.271867 21070 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:32:28.525791 21057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:28.526079 21057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:28.526211 21057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:28.533179 21057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:28.533267 21057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:28.533402 21057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:32:28.542557 21057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.193.keytab
--principal=oryx/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:34139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=36539
--tserver_master_addrs=127.6.46.254:35109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:28.543700 21057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:28.544970 21057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:28.558605 21093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:28.558642 21096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:28.558665 21101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:28.560072 21057 server_base.cc:1034] running on GCE node
I20260810 14:32:28.560624 21057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:28.561476 21057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:28.562690 21057 hybrid_clock.cc:648] HybridClock initialized: now 1786372348562638 us; error 47 us; skew 500 ppm
I20260810 14:32:28.562974 21057 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:28 dist-test-slave-q9h9 krb5kdc[20080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372348, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:28.565946 21057 init.cc:377] Logged in from keytab as oryx/127.6.46.193@KRBTEST.COM (short username oryx)
I20260810 14:32:28.567451 21057 webserver.cc:460] Webserver started at http://127.6.46.193:36539/ using document root <none> and password file <none>
I20260810 14:32:28.568082 21057 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:28.568197 21057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:28.573237 21057 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.001s
I20260810 14:32:28.575806 21109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:28.577052 21057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:32:28.577183 21057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "256be682e8d741d5b0ee963fdd26d9bb"
format_stamp: "Formatted at 2026-08-10 14:32:26 on dist-test-slave-q9h9"
server_key: "121de39b234b0458eecb0f788a3d79a5"
server_key_iv: "545a294a04e38b163b4ee020e23482f0"
server_key_version: "encryptionkey@0"
I20260810 14:32:28.578090 21057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:28.592073 21057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:28.595944 21057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:28.596186 21057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:28.597047 21057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:28.598313 21057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:28.598407 21057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:28.598507 21057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:28.598584 21057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:28.627765 21057 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:34139
I20260810 14:32:28.627830 21292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:34139 every 8 connection(s)
I20260810 14:32:28.629138 21057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:32:28.633713 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 21057
I20260810 14:32:28.634341 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:45229
--local_ip_for_outbound_sockets=127.6.46.194
--tserver_master_addrs=127.6.46.254:35109
--webserver_port=32827
--webserver_interface=127.6.46.194
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.46.194
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:32:28 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:28 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:28.647266 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:28.632164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:38867)
0810 14:32:28.633057 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:28.633110 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:28.634951 (+ 1841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:28.636346 (+ 1395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:28.636356 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:28.636906 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:28.637419 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:28.637436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:28.639113 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:28.639116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:28.640292 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:28.640304 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:28.641301 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:28.641309 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:28.641545 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:28.642206 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:28.642226 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:28.646159 (+ 3933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":366,"threads_started":1}
W20260810 14:32:28.648073 21296 heartbeater.cc:644] Failed to heartbeat to 127.6.46.254:35109 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.46.254:35109: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
W20260810 14:32:28.648134 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:28.632149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:38867 (local address 127.6.46.254:35109)
0810 14:32:28.633023 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:28.633039 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:28.633263 (+ 224us) server_negotiation.cc:407] Connection header received
0810 14:32:28.635481 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:28.635494 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:28.635808 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:28.636141 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:28.637639 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:28.638952 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:28.640436 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:28.641063 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:28.641233 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:28.646538 (+ 5305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:38867: BlockingRecv error: recv got EOF from 127.6.46.193:38867 (error 108)
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":380,"threads_started":1}
Aug 10 14:32:28 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:28 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:28.656559 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:28.648750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:53689)
0810 14:32:28.648935 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:28.648952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:28.649040 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:28.649357 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:28.649360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:28.649650 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:28.649787 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:28.649796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:28.651079 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:28.651083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:28.651985 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:28.651997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:28.652789 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:28.652793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:28.652917 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:28.653481 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:28.653499 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:28.656349 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 14:32:28.656807 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:28.648861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:53689 (local address 127.6.46.254:35109)
0810 14:32:28.649011 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:28.649017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:28.649039 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:32:28.649132 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:28.649137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:28.649216 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:28.649324 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:28.649919 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:28.650937 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:28.652135 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:28.652668 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:28.652775 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:28.656588 (+ 3813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:53689: BlockingRecv error: recv got EOF from 127.6.46.193:53689 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 14:32:28 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:28 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:28.665047 21299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:28.657179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:40413)
0810 14:32:28.657444 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:28.657472 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:28.657561 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:28.657865 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:28.657868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:28.658207 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:28.658402 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:28.658416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:28.659751 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:28.659755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:28.660662 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:28.660674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:28.661537 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:28.661541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:28.661683 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:28.662264 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:28.662282 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:28.664842 (+ 2560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162}
W20260810 14:32:28.665306 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:28.657276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:40413 (local address 127.6.46.254:35109)
0810 14:32:28.657454 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:28.657459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:28.657547 (+ 88us) server_negotiation.cc:407] Connection header received
0810 14:32:28.657667 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:28.657671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:28.657750 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:28.657846 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:28.658563 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:28.659579 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:28.660813 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:28.661404 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:28.661532 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:28.665069 (+ 3537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:40413: BlockingRecv error: recv got EOF from 127.6.46.193:40413 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 14:32:28.665437 21296 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:32:28.956413 21303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:28.956650 21303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:28.956768 21303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:28.963687 21303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:28.963778 21303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:28.963922 21303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:32:28.972990 21303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.194.keytab
--principal=oryx/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:45229
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=32827
--tserver_master_addrs=127.6.46.254:35109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:28.974171 21303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:28.975769 21303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:28.988394 21311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:28.988432 21313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:28.988454 21310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:28.989383 21303 server_base.cc:1034] running on GCE node
I20260810 14:32:28.989889 21303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:28.990768 21303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:28.991966 21303 hybrid_clock.cc:648] HybridClock initialized: now 1786372348991918 us; error 43 us; skew 500 ppm
I20260810 14:32:28.992249 21303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:28 dist-test-slave-q9h9 krb5kdc[20080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372348, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:28.995225 21303 init.cc:377] Logged in from keytab as oryx/127.6.46.194@KRBTEST.COM (short username oryx)
I20260810 14:32:28.996543 21303 webserver.cc:460] Webserver started at http://127.6.46.194:32827/ using document root <none> and password file <none>
I20260810 14:32:28.997154 21303 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:28.997280 21303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:29.002427 21303 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:32:29.004945 21335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:29.006193 21303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:32:29.006345 21303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "d769cfbeff2541069a44b6882100d263"
format_stamp: "Formatted at 2026-08-10 14:32:26 on dist-test-slave-q9h9"
server_key: "43cb816545e5ad3a385a021682dd7c31"
server_key_iv: "713d6b5987670e65abd58f2dd1672893"
server_key_version: "encryptionkey@0"
I20260810 14:32:29.007226 21303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:29.024608 21303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:29.028122 21303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:29.028364 21303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:29.029248 21303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:29.030530 21303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:29.030622 21303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:29.030733 21303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:29.030803 21303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:29.060822 21303 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:45229
I20260810 14:32:29.060882 21520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:45229 every 8 connection(s)
I20260810 14:32:29.062248 21303 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:32:29.072079 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 21303
I20260810 14:32:29.072674 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:45057
--local_ip_for_outbound_sockets=127.6.46.195
--tserver_master_addrs=127.6.46.254:35109
--webserver_port=44737
--webserver_interface=127.6.46.195
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.46.195
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:32:29.077347 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:29.064732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:35137 (local address 127.6.46.254:35109)
0810 14:32:29.064912 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:29.064917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:29.066062 (+ 1145us) server_negotiation.cc:407] Connection header received
0810 14:32:29.067835 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:29.067841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:29.067918 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:29.068012 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:29.069454 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.070245 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.071737 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.072143 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.072250 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.077162 (+ 4912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:35137: BlockingRecv error: recv got EOF from 127.6.46.194:35137 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:32:29.078138 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:29.065183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:35137)
0810 14:32:29.065894 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:29.065930 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:29.067577 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:29.068251 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:29.068259 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:29.068776 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:29.069285 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.069301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.070408 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.070412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.071593 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.071606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.072365 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.072372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.072602 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.073172 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:29.073192 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:29.077002 (+ 3810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":284,"threads_started":1}
W20260810 14:32:29.078871 21521 heartbeater.cc:644] Failed to heartbeat to 127.6.46.254:35109 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.46.254:35109: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:29.087145 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:29.079379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:47255)
0810 14:32:29.079539 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:29.079557 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:29.079644 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:29.079916 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:29.079920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:29.080236 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:29.080365 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.080372 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.081475 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.081479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.082719 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.082735 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.083571 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.083576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.083751 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.084275 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:29.084296 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:29.086973 (+ 2677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 14:32:29.087420 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:29.079496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:47255 (local address 127.6.46.254:35109)
0810 14:32:29.079647 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:29.079652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:29.079669 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:32:29.079738 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:29.079743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:29.079798 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:29.079872 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:29.080511 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.081338 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.082847 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.083408 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.083544 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.087203 (+ 3659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:47255: BlockingRecv error: recv got EOF from 127.6.46.194:47255 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:29.095207 21524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:29.087944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:53849)
0810 14:32:29.088086 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:29.088104 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:29.088183 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:29.088415 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:29.088417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:29.088667 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:29.088800 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.088807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.090084 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.090087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.090943 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.090953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.091777 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.091781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.091909 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.092401 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:29.092416 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:29.095009 (+ 2593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 14:32:29.095446 21521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:32:29.095842 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:29.088034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:53849 (local address 127.6.46.254:35109)
0810 14:32:29.088187 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:29.088191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:29.088209 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:32:29.088251 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:29.088255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:29.088311 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:29.088397 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:29.088944 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.089941 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.091081 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.091657 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.091785 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.095239 (+ 3454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:53849: BlockingRecv error: recv got EOF from 127.6.46.194:53849 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260810 14:32:29.392055 21526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:29.392302 21526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:29.392400 21526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:29.399164 21526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:29.399253 21526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:29.399363 21526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:32:29.408622 21526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/krb5kdc/oryx_127.6.46.195.keytab
--principal=oryx/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:45057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=44737
--tserver_master_addrs=127.6.46.254:35109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:29.409680 21526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:29.410984 21526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:29.423532 21532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:29.423605 21533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:29.423581 21535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:29.424381 21526 server_base.cc:1034] running on GCE node
I20260810 14:32:29.424939 21526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:29.425817 21526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:29.427011 21526 hybrid_clock.cc:648] HybridClock initialized: now 1786372349426942 us; error 70 us; skew 500 ppm
I20260810 14:32:29.427295 21526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372349, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:29.430275 21526 init.cc:377] Logged in from keytab as oryx/127.6.46.195@KRBTEST.COM (short username oryx)
I20260810 14:32:29.431578 21526 webserver.cc:460] Webserver started at http://127.6.46.195:44737/ using document root <none> and password file <none>
I20260810 14:32:29.432196 21526 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:29.432322 21526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:29.437325 21526 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:32:29.439831 21548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:29.441052 21526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:32:29.441200 21526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "9264f29486354e688339282e2f7fc67e"
format_stamp: "Formatted at 2026-08-10 14:32:27 on dist-test-slave-q9h9"
server_key: "aacf8377a284cdb430b91817c9fd6546"
server_key_iv: "655963f3aae406df4a53c6a837898a26"
server_key_version: "encryptionkey@0"
I20260810 14:32:29.442099 21526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:29.465149 21526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:29.468762 21526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:29.468997 21526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:29.469913 21526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:29.471182 21526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:29.471287 21526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:29.471392 21526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:29.471467 21526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:29.499971 21526 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:45057
I20260810 14:32:29.500039 21737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:45057 every 8 connection(s)
I20260810 14:32:29.501281 21526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:32:29.509598 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 21526
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 14:32:29.516362 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:29.503708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:59235 (local address 127.6.46.254:35109)
0810 14:32:29.503888 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:29.503892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:29.504934 (+ 1042us) server_negotiation.cc:407] Connection header received
0810 14:32:29.506707 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:29.506712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:29.506783 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:29.506867 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:29.508245 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.508955 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.510639 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.511348 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.511482 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.516149 (+ 4667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:59235: BlockingRecv error: recv got EOF from 127.6.46.195:59235 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:32:29.517055 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:29.504025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:59235)
0810 14:32:29.504756 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:29.504791 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:29.506461 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:29.507066 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:29.507073 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:29.507557 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:29.508086 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.508102 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.509076 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.509079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.510491 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.510506 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.511575 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.511581 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.511766 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.512380 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:29.512407 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:29.515952 (+ 3545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":302,"threads_started":1}
W20260810 14:32:29.517692 21738 heartbeater.cc:644] Failed to heartbeat to 127.6.46.254:35109 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.46.254:35109: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:29.526314 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:29.518321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:38737)
0810 14:32:29.518565 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:29.518582 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:29.518688 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:29.518958 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:29.518963 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:29.519347 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:29.519507 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.519517 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.520509 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.520513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.521790 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.521806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.522576 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.522583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.522750 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.523202 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:29.523224 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:29.526065 (+ 2841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137}
W20260810 14:32:29.526427 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:29.518400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:38737 (local address 127.6.46.254:35109)
0810 14:32:29.518528 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:29.518531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:29.518645 (+ 114us) server_negotiation.cc:407] Connection header received
0810 14:32:29.518781 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:29.518784 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:29.518832 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:29.518902 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:29.519621 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.520364 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.521943 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.522424 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.522551 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.526239 (+ 3688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:38737: BlockingRecv error: recv got EOF from 127.6.46.195:38737 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:32:29.526948 21743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:29.510678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51010 (local address 127.6.46.254:35109)
0810 14:32:29.511284 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:29.511290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:29.511722 (+ 432us) server_negotiation.cc:407] Connection header received
0810 14:32:29.511925 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:29.511929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:29.511982 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:29.512068 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:29.513629 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.514475 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.515771 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.516405 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.516529 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.519853 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:29.519861 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:29.519876 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:29.519912 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:29.523296 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:29.523955 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:29.523963 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:29.523971 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:29.524205 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:29.524617 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:29.524621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:29.524622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:29.525069 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:29.525341 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:29.526373 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0810 14:32:29.526694 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":280,"threads_started":1}
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:29.534360 21741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:29.527056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:59407)
0810 14:32:29.527208 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:29.527225 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:29.527317 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:29.527725 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:29.527729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:29.528068 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:29.528228 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.528236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.529491 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.529496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.530639 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.530653 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.531296 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.531300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.531418 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.531881 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:29.531896 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:29.534175 (+ 2279us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 14:32:29.534475 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:29.527278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:59407 (local address 127.6.46.254:35109)
0810 14:32:29.527489 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:29.527493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:29.527507 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:32:29.527540 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:29.527543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:29.527586 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:29.527660 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:29.528367 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.529355 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.530794 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.531188 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.531283 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.534312 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:59407: BlockingRecv error: recv got EOF from 127.6.46.195:59407 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260810 14:32:29.534637 21738 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:29 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:29.674041 21745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:29.665942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:46339)
0810 14:32:29.666421 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:29.666438 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:29.666547 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:29.666811 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:29.666814 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:29.667248 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:29.667438 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.667447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.668361 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.668365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.669403 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:29.669416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.670398 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.670405 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:29.670540 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.671098 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:29.671115 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:29.673794 (+ 2679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":266,"threads_started":1}
W20260810 14:32:29.674230 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:29.666029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:46339 (local address 127.6.46.254:35109)
0810 14:32:29.666256 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:29.666263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:29.666517 (+ 254us) server_negotiation.cc:407] Connection header received
0810 14:32:29.666637 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:29.666640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:29.666701 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:29.666778 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:29.667549 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.668256 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.669543 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:29.670207 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:29.670332 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:29.673989 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:46339: BlockingRecv error: recv got EOF from 127.6.46.193:46339 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 14:32:30 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:30 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:30.104409 21746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:30.095999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:55577)
0810 14:32:30.096517 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:30.096533 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:30.096670 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:30.096990 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:30.096994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:30.097484 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:30.097740 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:30.097754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:30.098996 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:30.099000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:30.099952 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:30.099964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:30.100675 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:30.100682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:30.100834 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:30.101327 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:30.101344 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:30.104178 (+ 2834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":266,"threads_started":1}
W20260810 14:32:30.104585 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:30.096157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:55577 (local address 127.6.46.254:35109)
0810 14:32:30.096401 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:30.096407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:30.096613 (+ 206us) server_negotiation.cc:407] Connection header received
0810 14:32:30.096784 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:30.096789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:30.096851 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:30.096931 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:30.097930 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:30.098864 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:30.100070 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:30.100496 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:30.100586 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:30.104372 (+ 3786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:55577: BlockingRecv error: recv got EOF from 127.6.46.194:55577 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 10 14:32:30 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:30 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:30.546828 21747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:30.535190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:33355)
0810 14:32:30.535736 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:30.535752 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:30.535880 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:30.536221 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:30.536225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:30.536689 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:30.536896 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:30.536909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:30.537936 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:30.537941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:30.538936 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:30.538949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:30.539766 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:30.539772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:30.539926 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:30.543633 (+ 3707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:30.543652 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:30.546513 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":296,"threads_started":1}
W20260810 14:32:30.546960 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:30.535300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:33355 (local address 127.6.46.254:35109)
0810 14:32:30.535513 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:30.535519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:30.535840 (+ 321us) server_negotiation.cc:407] Connection header received
0810 14:32:30.535994 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:30.535999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:30.536062 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:30.536161 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:30.537053 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:30.537785 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:30.539091 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:30.539585 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:30.539695 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:30.546741 (+ 7046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:33355: BlockingRecv error: recv got EOF from 127.6.46.195:33355 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 14:32:30 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:30 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:30.683308 21749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:30.674966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:44147)
0810 14:32:30.675549 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:30.675566 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:30.675684 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:30.675964 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:30.675967 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:30.676348 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:30.676505 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:30.676514 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:30.677506 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:30.677509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:30.678454 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:30.678466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:30.679355 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:30.679360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:30.679511 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:30.680181 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:30.680199 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:30.683065 (+ 2866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":317,"threads_started":1}
W20260810 14:32:30.683459 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:30.675100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:44147 (local address 127.6.46.254:35109)
0810 14:32:30.675290 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:30.675294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:30.675677 (+ 383us) server_negotiation.cc:407] Connection header received
0810 14:32:30.675781 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:30.675785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:30.675848 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:30.675928 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:30.676629 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:30.677382 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:30.678616 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:30.679183 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:30.679314 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:30.683241 (+ 3927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:44147: BlockingRecv error: recv got EOF from 127.6.46.193:44147 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 14:32:31 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:31 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:31.113590 21756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:31.105295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:60777)
0810 14:32:31.105799 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:31.105815 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:31.105924 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:31.106352 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:31.106356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:31.106745 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:31.106889 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:31.106898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:31.107929 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:31.107933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:31.108852 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:31.108869 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:31.109824 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:31.109831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:31.109955 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:31.110564 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:31.110582 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:31.113268 (+ 2686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":305,"threads_started":1}
W20260810 14:32:31.113713 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:31.105377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:60777 (local address 127.6.46.254:35109)
0810 14:32:31.105625 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:31.105631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:31.105901 (+ 270us) server_negotiation.cc:407] Connection header received
0810 14:32:31.106139 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:31.106144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:31.106212 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:31.106307 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:31.107035 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:31.107790 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:31.109026 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:31.109660 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:31.109786 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:31.113486 (+ 3700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:60777: BlockingRecv error: recv got EOF from 127.6.46.194:60777 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Aug 10 14:32:31 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:31 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:31.555871 21757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:31.547656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:54365)
0810 14:32:31.548176 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:31.548193 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:31.548317 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:31.548593 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:31.548596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:31.549015 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:31.549163 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:31.549173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:31.550276 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:31.550280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:31.551426 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:31.551445 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:31.552272 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:31.552279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:31.552401 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:31.552930 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:31.552947 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:31.555630 (+ 2683us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":281,"threads_started":1}
W20260810 14:32:31.556030 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:31.547793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:54365 (local address 127.6.46.254:35109)
0810 14:32:31.547964 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:31.547969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:31.548259 (+ 290us) server_negotiation.cc:407] Connection header received
0810 14:32:31.548408 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:31.548412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:31.548482 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:31.548560 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:31.549302 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:31.550151 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:31.551564 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:31.552100 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:31.552215 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:31.555801 (+ 3586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:54365: BlockingRecv error: recv got EOF from 127.6.46.195:54365 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 14:32:31 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:31 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:31.691784 21758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:31.684089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:35851)
0810 14:32:31.684570 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:31.684587 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:31.684701 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:31.684978 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:31.684981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:31.685344 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:31.685511 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:31.685520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:31.686455 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:31.686459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:31.687337 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:31.687348 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:31.687912 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:31.687915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:31.688030 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:31.688637 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:31.688654 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:31.691422 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":262,"threads_started":1}
W20260810 14:32:31.691927 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:31.684173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:35851 (local address 127.6.46.254:35109)
0810 14:32:31.684339 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:31.684343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:31.684659 (+ 316us) server_negotiation.cc:407] Connection header received
0810 14:32:31.684794 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:31.684797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:31.684857 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:31.684937 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:31.685623 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:31.686337 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:31.687441 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:31.687812 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:31.687893 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:31.691664 (+ 3771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:35851: BlockingRecv error: recv got EOF from 127.6.46.193:35851 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 14:32:32 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:32 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:32.122936 21761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:32.114567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:45817)
0810 14:32:32.115044 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:32.115059 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:32.115204 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:32.115489 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:32.115492 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:32.115843 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:32.116010 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:32.116019 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:32.117138 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:32.117143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:32.118096 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:32.118109 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:32.118812 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:32.118817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:32.118947 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:32.119562 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:32.119584 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:32.122589 (+ 3005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":267,"threads_started":1}
W20260810 14:32:32.123075 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:32.114702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:45817 (local address 127.6.46.254:35109)
0810 14:32:32.114951 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:32.114957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:32.115164 (+ 207us) server_negotiation.cc:407] Connection header received
0810 14:32:32.115297 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:32.115300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:32.115361 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:32.115439 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:32.116176 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:32.116984 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:32.118246 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:32.118668 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:32.118767 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:32.122849 (+ 4082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:45817: BlockingRecv error: recv got EOF from 127.6.46.194:45817 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Aug 10 14:32:32 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:32 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:32.565063 21762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:32.556739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:55177)
0810 14:32:32.557437 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:32.557458 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:32.557589 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:32.557878 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:32.557889 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:32.558271 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:32.558444 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:32.558453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:32.559405 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:32.559411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:32.560344 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:32.560357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:32.560941 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:32.560945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:32.561069 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:32.561654 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:32.561673 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:32.564723 (+ 3050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":486,"threads_started":1}
W20260810 14:32:32.565172 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:32.556818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:55177 (local address 127.6.46.254:35109)
0810 14:32:32.557059 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:32.557066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:32.557540 (+ 474us) server_negotiation.cc:407] Connection header received
0810 14:32:32.557682 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:32.557686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:32.557764 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:32.557841 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:32.558558 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:32.559263 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:32.560454 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:32.560831 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:32.560916 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:32.564940 (+ 4024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:55177: BlockingRecv error: recv got EOF from 127.6.46.195:55177 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 10 14:32:32 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:32 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:32.700816 21763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:32.692667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:47389)
0810 14:32:32.693182 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:32.693203 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:32.693350 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:32.693711 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:32.693716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:32.694230 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:32.694412 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:32.694422 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:32.695397 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:32.695400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:32.696354 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:32.696365 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:32.697070 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:32.697075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:32.697220 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:32.697765 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:32.697782 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:32.700511 (+ 2729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
W20260810 14:32:32.700966 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:32.692760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:47389 (local address 127.6.46.254:35109)
0810 14:32:32.692978 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:32.693001 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:32.693299 (+ 298us) server_negotiation.cc:407] Connection header received
0810 14:32:32.693486 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:32.693491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:32.693572 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:32.693681 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:32.694548 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:32.695298 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:32.696503 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:32.696907 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:32.697024 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:32.700736 (+ 3712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:47389: BlockingRecv error: recv got EOF from 127.6.46.193:47389 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 14:32:33 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:33 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:33.132233 21765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:33.123925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:54207)
0810 14:32:33.124389 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:33.124406 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:33.124518 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:33.124798 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:33.124801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:33.125177 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:33.125327 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:33.125335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:33.126571 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:33.126577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:33.127548 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:33.127561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:33.128278 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:33.128285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:33.128472 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:33.129053 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:33.129071 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:33.131922 (+ 2851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":255,"threads_started":1}
W20260810 14:32:33.132361 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:33.124001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:54207 (local address 127.6.46.254:35109)
0810 14:32:33.124205 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:33.124212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:33.124481 (+ 269us) server_negotiation.cc:407] Connection header received
0810 14:32:33.124608 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:33.124611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:33.124682 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:33.124756 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:33.125461 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:33.126448 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:33.127706 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:33.128117 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:33.128227 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:33.132147 (+ 3920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:54207: BlockingRecv error: recv got EOF from 127.6.46.194:54207 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 14:32:33 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:33 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:33.574126 21766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:33.566093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:34681)
0810 14:32:33.566555 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:33.566571 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:33.566693 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:33.566954 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:33.566957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:33.567327 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:33.567477 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:33.567486 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:33.568466 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:33.568470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:33.569433 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:33.569444 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:33.570078 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:33.570082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:33.570214 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:33.570784 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:33.570807 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:33.573793 (+ 2986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":249,"threads_started":1}
W20260810 14:32:33.574280 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:33.566130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:34681 (local address 127.6.46.254:35109)
0810 14:32:33.566334 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:33.566342 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:33.566645 (+ 303us) server_negotiation.cc:407] Connection header received
0810 14:32:33.566782 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:33.566786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:33.566847 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:33.566921 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:33.567603 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:33.568342 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:33.569539 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:33.569932 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:33.570044 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:33.574008 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:34681: BlockingRecv error: recv got EOF from 127.6.46.195:34681 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 14:32:33 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:33 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:33.709251 21767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:33.701716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:55133)
0810 14:32:33.702245 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:33.702261 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:33.702375 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:33.702629 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:33.702632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:33.703063 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:33.703236 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:33.703246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:33.704180 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:33.704184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:33.705086 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:33.705097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:33.705731 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:33.705735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:33.705848 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:33.706406 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:33.706423 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:33.709009 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":243,"threads_started":1}
W20260810 14:32:33.709460 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:33.701789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:55133 (local address 127.6.46.254:35109)
0810 14:32:33.701971 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:33.701975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:33.702332 (+ 357us) server_negotiation.cc:407] Connection header received
0810 14:32:33.702463 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:33.702466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:33.702525 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:33.702600 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:33.703349 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:33.704078 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:33.705197 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:33.705628 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:33.705713 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:33.709233 (+ 3520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:55133: BlockingRecv error: recv got EOF from 127.6.46.193:55133 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 14:32:34 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:34 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:34.141458 21768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:34.133177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:40431)
0810 14:32:34.133683 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:34.133699 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:34.133809 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:34.134096 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:34.134099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:34.134512 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:34.134677 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:34.134685 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:34.135673 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:34.135678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:34.136628 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:34.136641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:34.137511 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:34.137518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:34.137697 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:34.138386 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:34.138410 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:34.141125 (+ 2715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":242,"threads_started":1}
W20260810 14:32:34.141602 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:34.133368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:40431 (local address 127.6.46.254:35109)
0810 14:32:34.133564 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:34.133570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:34.133770 (+ 200us) server_negotiation.cc:407] Connection header received
0810 14:32:34.133896 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:34.133900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:34.133958 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:34.134058 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:34.134810 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:34.135530 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:34.136759 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:34.137344 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:34.137453 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:34.141359 (+ 3906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:40431: BlockingRecv error: recv got EOF from 127.6.46.194:40431 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 10 14:32:34 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:34 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:34.586906 21769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:34.575004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:57109)
0810 14:32:34.575512 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:34.575528 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:34.575654 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:34.575963 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:34.575968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:34.576394 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:34.576571 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:34.576581 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:34.577854 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:34.577861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:34.578782 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:34.578795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:34.579493 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:34.579499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:34.579659 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:34.583883 (+ 4224us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:34.583901 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:34.586571 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":250,"threads_started":1}
W20260810 14:32:34.587015 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:34.575080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:57109 (local address 127.6.46.254:35109)
0810 14:32:34.575257 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:34.575264 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:34.575615 (+ 351us) server_negotiation.cc:407] Connection header received
0810 14:32:34.575771 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:34.575775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:34.575834 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:34.575911 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:34.576725 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:34.577715 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:34.578926 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:34.579349 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:34.579461 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:34.586786 (+ 7325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:57109: BlockingRecv error: recv got EOF from 127.6.46.195:57109 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 14:32:34 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:34 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:34.717867 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:34.710167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:34557)
0810 14:32:34.710714 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:34.710731 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:34.710860 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:34.711143 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:34.711146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:34.711553 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:34.711711 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:34.711721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:34.712677 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:34.712681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:34.713574 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:34.713585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:34.714261 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:34.714265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:34.714385 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:34.714958 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:34.714975 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:34.717654 (+ 2679us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":304,"threads_started":1}
W20260810 14:32:34.718081 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:34.710229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:34557 (local address 127.6.46.254:35109)
0810 14:32:34.710386 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:34.710391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:34.710797 (+ 406us) server_negotiation.cc:407] Connection header received
0810 14:32:34.710952 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:34.710955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:34.711017 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:34.711093 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:34.711831 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:34.712572 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:34.713687 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:34.714141 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:34.714243 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:34.717827 (+ 3584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:34557: BlockingRecv error: recv got EOF from 127.6.46.193:34557 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 14:32:35 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:35 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:35.150584 21774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:35.142434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:42337)
0810 14:32:35.142924 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:35.142939 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:35.143072 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:35.143392 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:35.143398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:35.143796 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:35.143955 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:35.143963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:35.145000 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:35.145005 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:35.146121 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:35.146135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:35.146951 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:35.146957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:35.147123 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:35.147657 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:35.147673 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:35.150332 (+ 2659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":253,"threads_started":1}
W20260810 14:32:35.150794 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:35.142520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:42337 (local address 127.6.46.254:35109)
0810 14:32:35.142708 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:35.142714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:35.143023 (+ 309us) server_negotiation.cc:407] Connection header received
0810 14:32:35.143200 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:35.143204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:35.143265 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:35.143343 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:35.144098 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:35.144860 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:35.146258 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:35.146795 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:35.146899 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:35.150550 (+ 3651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:42337: BlockingRecv error: recv got EOF from 127.6.46.194:42337 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 14:32:35 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:35 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:35.598266 21775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:35.587820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:56293)
0810 14:32:35.588312 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:35.588329 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:35.588447 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:35.588720 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:35.588723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:35.589101 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:35.589257 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:35.589265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:35.590261 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:35.590265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:35.591167 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:35.591179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:35.591992 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:35.591996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:35.592117 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:35.595316 (+ 3199us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:35.595334 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:35.597991 (+ 2657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":270,"threads_started":1}
W20260810 14:32:35.598424 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:35.587901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:56293 (local address 127.6.46.254:35109)
0810 14:32:35.588108 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:35.588115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:35.588409 (+ 294us) server_negotiation.cc:407] Connection header received
0810 14:32:35.588548 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:35.588551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:35.588615 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:35.588684 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:35.589367 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:35.590149 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:35.591360 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:35.591863 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:35.591961 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:35.598212 (+ 6251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:56293: BlockingRecv error: recv got EOF from 127.6.46.195:56293 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 14:32:35 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:35 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:35.726962 21782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:35.718721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:56853)
0810 14:32:35.719251 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:35.719267 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:35.719405 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:35.719713 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:35.719718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:35.720171 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:35.720343 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:35.720353 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:35.721368 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:35.721372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:35.722321 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:35.722333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:35.723005 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:35.723010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:35.723126 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:35.723667 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:35.723685 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:35.726719 (+ 3034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":232,"threads_started":1}
W20260810 14:32:35.727103 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:35.718817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:56853 (local address 127.6.46.254:35109)
0810 14:32:35.718999 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:35.719005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:35.719358 (+ 353us) server_negotiation.cc:407] Connection header received
0810 14:32:35.719518 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:35.719524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:35.719585 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:35.719677 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:35.720496 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:35.721223 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:35.722455 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:35.722883 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:35.722980 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:35.726913 (+ 3933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:56853: BlockingRecv error: recv got EOF from 127.6.46.193:56853 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 14:32:36 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:36 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:36.160069 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:36.151484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:35761)
0810 14:32:36.152033 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:36.152049 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:36.152168 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:36.152441 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:36.152445 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:36.152853 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:36.153020 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:36.153029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:36.154406 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:36.154413 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:36.155494 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:36.155510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:36.156139 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:36.156143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:36.156266 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:36.156817 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:36.156836 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:36.159821 (+ 2985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":254,"threads_started":1}
W20260810 14:32:36.160238 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:36.151542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:35761 (local address 127.6.46.254:35109)
0810 14:32:36.151741 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:36.151747 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:36.152128 (+ 381us) server_negotiation.cc:407] Connection header received
0810 14:32:36.152271 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:36.152275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:36.152337 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:36.152412 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:36.153178 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:36.154237 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:36.155633 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:36.156014 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:36.156104 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:36.160006 (+ 3902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:35761: BlockingRecv error: recv got EOF from 127.6.46.194:35761 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 14:32:36 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:36 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:36.607227 21784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:36.599120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:59373)
0810 14:32:36.599678 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:36.599696 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:36.599810 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:36.600136 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:36.600140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:36.600539 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:36.600731 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:36.600740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:36.601692 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:36.601696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:36.602689 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:36.602701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:36.603427 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:36.603432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:36.603550 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:36.604085 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:36.604103 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:36.606885 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":253,"threads_started":1}
W20260810 14:32:36.607336 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:36.599245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:59373 (local address 127.6.46.254:35109)
0810 14:32:36.599404 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:36.599409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:36.599761 (+ 352us) server_negotiation.cc:407] Connection header received
0810 14:32:36.599898 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:36.599901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:36.599960 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:36.600043 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:36.600847 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:36.601544 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:36.602825 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:36.603296 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:36.603389 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:36.607123 (+ 3734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:59373: BlockingRecv error: recv got EOF from 127.6.46.195:59373 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 14:32:36 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:36 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:36.735677 21785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:36.727802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:47811)
0810 14:32:36.728311 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:36.728329 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:36.728452 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:36.728721 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:36.728725 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:36.729128 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:36.729277 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:36.729286 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:36.730253 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:36.730257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:36.731178 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:36.731189 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:36.731851 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:36.731856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:36.731970 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:36.732509 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:36.732525 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:36.735440 (+ 2915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":278,"threads_started":1}
W20260810 14:32:36.735847 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:36.727980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:47811 (local address 127.6.46.254:35109)
0810 14:32:36.728148 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:36.728153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:36.728395 (+ 242us) server_negotiation.cc:407] Connection header received
0810 14:32:36.728540 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:36.728544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:36.728612 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:36.728690 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:36.729389 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:36.730120 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:36.731286 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:36.731715 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:36.731803 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:36.735630 (+ 3827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:47811: BlockingRecv error: recv got EOF from 127.6.46.193:47811 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 14:32:37 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:37 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:37.168987 21787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:37.160935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:54951)
0810 14:32:37.161459 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:37.161475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:37.161613 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:37.161922 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:37.161927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:37.162376 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:37.162529 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:37.162538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:37.163704 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:37.163709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:37.164613 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:37.164625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:37.165362 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:37.165369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:37.165523 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:37.166105 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:37.166122 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:37.168671 (+ 2549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":263,"threads_started":1}
W20260810 14:32:37.169152 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:37.161057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:54951 (local address 127.6.46.254:35109)
0810 14:32:37.161267 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:37.161273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:37.161554 (+ 281us) server_negotiation.cc:407] Connection header received
0810 14:32:37.161727 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:37.161731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:37.161790 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:37.161869 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:37.162676 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:37.163554 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:37.164751 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:37.165185 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:37.165295 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:37.168924 (+ 3629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:54951: BlockingRecv error: recv got EOF from 127.6.46.194:54951 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Aug 10 14:32:37 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:37 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:37.616313 21792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:37.608152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:58113)
0810 14:32:37.608678 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:37.608694 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:37.608826 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:37.609130 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:37.609135 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:37.609559 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:37.609725 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:37.609735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:37.610855 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:37.610859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:37.611854 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:37.611868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:37.612602 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:37.612609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:37.612762 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:37.613353 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:37.613372 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:37.616014 (+ 2642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":264,"threads_started":1}
W20260810 14:32:37.616465 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:37.608252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:58113 (local address 127.6.46.254:35109)
0810 14:32:37.608459 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:37.608466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:37.608776 (+ 310us) server_negotiation.cc:407] Connection header received
0810 14:32:37.608938 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:37.608943 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:37.609003 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:37.609083 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:37.609871 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:37.610699 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:37.611996 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:37.612424 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:37.612532 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:37.616240 (+ 3708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:58113: BlockingRecv error: recv got EOF from 127.6.46.195:58113 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 14:32:37 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:37 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:37.744670 21793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:37.736555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:37795)
0810 14:32:37.737130 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:37.737152 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:37.737290 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:37.737620 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:37.737625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:37.738122 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:37.738326 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:37.738340 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:37.739353 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:37.739357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:37.740278 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:37.740290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:37.740919 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:37.740922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:37.741036 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:37.741594 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:37.741612 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:37.744335 (+ 2723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":310,"threads_started":1}
W20260810 14:32:37.744781 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:37.736614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:37795 (local address 127.6.46.254:35109)
0810 14:32:37.736803 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:37.736811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:37.737229 (+ 418us) server_negotiation.cc:407] Connection header received
0810 14:32:37.737419 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:37.737423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:37.737484 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:37.737564 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:37.738478 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:37.739232 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:37.740386 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:37.740804 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:37.740897 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:37.744552 (+ 3655us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:37795: BlockingRecv error: recv got EOF from 127.6.46.193:37795 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 14:32:38 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:38 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:38.178298 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:38.170053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:54179)
0810 14:32:38.170558 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:38.170573 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:38.170712 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:38.171030 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:38.171034 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:38.171443 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:38.171621 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:38.171630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:38.172682 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:38.172686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:38.173644 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:38.173656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:38.174403 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:38.174409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:38.174570 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:38.175175 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:38.175192 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:38.177937 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":266,"threads_started":1}
W20260810 14:32:38.178422 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:38.170106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:54179 (local address 127.6.46.254:35109)
0810 14:32:38.170291 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:38.170298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:38.170664 (+ 366us) server_negotiation.cc:407] Connection header received
0810 14:32:38.170823 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:38.170828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:38.170896 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:38.170975 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:38.171784 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:38.172544 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:38.173786 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:38.174256 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:38.174365 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:38.178179 (+ 3814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:54179: BlockingRecv error: recv got EOF from 127.6.46.194:54179 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 14:32:38 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:38 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:38.625440 21797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:38.617297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:50533)
0810 14:32:38.617819 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:38.617837 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:38.617973 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:38.618363 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:38.618368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:38.618790 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:38.618960 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:38.618970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:38.619995 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:38.619999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:38.620992 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:38.621010 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:38.621712 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:38.621719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:38.621888 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:38.622473 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:38.622491 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:38.625115 (+ 2624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":278,"threads_started":1}
W20260810 14:32:38.625542 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:38.617333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:50533 (local address 127.6.46.254:35109)
0810 14:32:38.617505 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:38.617509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:38.617925 (+ 416us) server_negotiation.cc:407] Connection header received
0810 14:32:38.618144 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:38.618149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:38.618213 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:38.618299 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:38.619116 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:38.619848 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:38.621144 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:38.621562 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:38.621669 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:38.625331 (+ 3662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:50533: BlockingRecv error: recv got EOF from 127.6.46.195:50533 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 14:32:38 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:38 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:38.753773 21799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:38.745647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:38057)
0810 14:32:38.746209 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:38.746229 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:38.746361 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:38.746701 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:38.746706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:38.747102 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:38.747259 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:38.747269 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:38.748333 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:38.748339 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:38.749318 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:38.749331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:38.750145 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:38.750152 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:38.750318 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:38.750881 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:38.750899 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:38.753457 (+ 2558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":339,"threads_started":1}
W20260810 14:32:38.753916 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:38.745684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:38057 (local address 127.6.46.254:35109)
0810 14:32:38.745910 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:38.745916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:38.746301 (+ 385us) server_negotiation.cc:407] Connection header received
0810 14:32:38.746472 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:38.746478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:38.746565 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:38.746642 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:38.747404 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:38.748203 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:38.749456 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:38.749968 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:38.750108 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:38.753681 (+ 3573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:38057: BlockingRecv error: recv got EOF from 127.6.46.193:38057 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 14:32:39 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:39 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:39.188014 21801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:39.179286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:51143)
0810 14:32:39.179748 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:39.179765 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:39.179895 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:39.180199 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:39.180204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:39.180614 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:39.180775 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:39.180785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:39.181856 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:39.181860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:39.182971 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:39.182985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:39.183960 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:39.183966 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:39.184145 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:39.184677 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:39.184694 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:39.187718 (+ 3024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":240,"threads_started":1}
W20260810 14:32:39.188217 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:39.179349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:51143 (local address 127.6.46.254:35109)
0810 14:32:39.179524 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:39.179531 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:39.179850 (+ 319us) server_negotiation.cc:407] Connection header received
0810 14:32:39.180007 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:39.180011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:39.180073 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:39.180152 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:39.180925 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:39.181711 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:39.183147 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:39.183818 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:39.183947 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:39.187952 (+ 4005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:51143: BlockingRecv error: recv got EOF from 127.6.46.194:51143 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 14:32:39 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:39 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:39.637398 21802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:39.626522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:58021)
0810 14:32:39.627162 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:39.627181 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:39.627298 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:39.627659 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:39.627664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:39.628176 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:39.628372 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:39.628386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:39.629824 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:39.629831 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:39.631348 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:39.631368 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:39.632725 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:39.632733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:39.632916 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:39.633672 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:39.633715 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:39.637100 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":395,"threads_started":1}
W20260810 14:32:39.637578 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:39.626690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:58021 (local address 127.6.46.254:35109)
0810 14:32:39.626940 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:39.626945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:39.627262 (+ 317us) server_negotiation.cc:407] Connection header received
0810 14:32:39.627418 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:39.627423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:39.627506 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:39.627618 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:39.628574 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:39.629655 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:39.631540 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:39.632171 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:39.632273 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:39.637340 (+ 5067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:58021: BlockingRecv error: recv got EOF from 127.6.46.195:58021 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Aug 10 14:32:39 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:39 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:39.762559 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:39.754712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:42621)
0810 14:32:39.755185 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:39.755200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:39.755328 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:39.755620 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:39.755625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:39.756049 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:39.756234 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:39.756244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:39.757242 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:39.757246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:39.758168 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:39.758180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:39.758879 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:39.758884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:39.759011 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:39.759534 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:39.759550 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:39.762272 (+ 2722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":244,"threads_started":1}
W20260810 14:32:39.762746 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:39.754767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:42621 (local address 127.6.46.254:35109)
0810 14:32:39.754972 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:39.754978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:39.755281 (+ 303us) server_negotiation.cc:407] Connection header received
0810 14:32:39.755438 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:39.755444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:39.755503 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:39.755580 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:39.756375 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:39.757110 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:39.758302 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:39.758724 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:39.758856 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:39.762470 (+ 3614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:42621: BlockingRecv error: recv got EOF from 127.6.46.193:42621 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 14:32:40 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:40 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:40.197014 21804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:40.188972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:41723)
0810 14:32:40.189461 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:40.189476 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:40.189612 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:40.189930 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:40.189935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:40.190385 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:40.190576 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:40.190586 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:40.191672 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:40.191678 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:40.192555 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:40.192567 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:40.193278 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:40.193284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:40.193442 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:40.194010 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:40.194063 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:40.196773 (+ 2710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":261,"threads_started":1}
W20260810 14:32:40.197157 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:40.189034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:41723 (local address 127.6.46.254:35109)
0810 14:32:40.189233 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:40.189239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:40.189556 (+ 317us) server_negotiation.cc:407] Connection header received
0810 14:32:40.189725 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:40.189731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:40.189801 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:40.189881 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:40.190730 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:40.191525 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:40.192698 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:40.193133 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:40.193238 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:40.196960 (+ 3722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:41723: BlockingRecv error: recv got EOF from 127.6.46.194:41723 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 14:32:40 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:40 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:40.649890 21809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:40.638458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:44089)
0810 14:32:40.638939 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:40.638955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:40.639064 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:40.639329 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:40.639332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:40.639738 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:40.639913 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:40.639922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:40.640916 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:40.640921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:40.641824 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:40.641836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:40.642516 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:40.642520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:40.642644 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:40.646330 (+ 3686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:40.646350 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:40.649591 (+ 3241us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":252,"threads_started":1}
W20260810 14:32:40.649976 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:40.638529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:44089 (local address 127.6.46.254:35109)
0810 14:32:40.638677 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:40.638681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:40.639016 (+ 335us) server_negotiation.cc:407] Connection header received
0810 14:32:40.639166 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:40.639170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:40.639230 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:40.639300 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:40.640047 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:40.640794 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:40.641943 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:40.642399 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:40.642492 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:40.649788 (+ 7296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:44089: BlockingRecv error: recv got EOF from 127.6.46.195:44089 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Aug 10 14:32:40 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:40 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:40.771456 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:40.763516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:53769)
0810 14:32:40.764083 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:40.764101 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:40.764213 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:40.764551 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:40.764557 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:40.765043 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:40.765266 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:40.765280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:40.766339 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:40.766343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:40.767244 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:40.767255 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:40.767878 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:40.767883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:40.768002 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:40.768591 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:40.768608 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:40.771223 (+ 2615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":304,"threads_started":1}
W20260810 14:32:40.771606 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:40.763562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:53769 (local address 127.6.46.254:35109)
0810 14:32:40.763768 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:40.763775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:40.764179 (+ 404us) server_negotiation.cc:407] Connection header received
0810 14:32:40.764319 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:40.764326 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:40.764387 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:40.764490 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:40.765433 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:40.766232 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:40.767347 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:40.767778 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:40.767864 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:40.771405 (+ 3541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:53769: BlockingRecv error: recv got EOF from 127.6.46.193:53769 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 14:32:41 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:41 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:41.205714 21813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:41.197861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:59397)
0810 14:32:41.198436 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:41.198453 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:41.198570 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:41.198844 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:41.198847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:41.199263 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:41.199427 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:41.199436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:41.200384 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:41.200388 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:41.201297 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:41.201310 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:41.201952 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:41.201956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:41.202115 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:41.202687 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:41.202705 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:41.205474 (+ 2769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":325,"threads_started":1}
W20260810 14:32:41.205847 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:41.197929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:59397 (local address 127.6.46.254:35109)
0810 14:32:41.198139 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:41.198144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:41.198532 (+ 388us) server_negotiation.cc:407] Connection header received
0810 14:32:41.198660 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:41.198663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:41.198724 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:41.198801 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:41.199559 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:41.200274 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:41.201406 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:41.201846 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:41.201936 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:41.205663 (+ 3727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:59397: BlockingRecv error: recv got EOF from 127.6.46.194:59397 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 14:32:41 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:41 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:41.658880 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:41.650815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:34443)
0810 14:32:41.651325 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:41.651341 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:41.651475 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:41.651803 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:41.651809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:41.652224 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:41.652401 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:41.652411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:41.653425 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:41.653429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:41.654367 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:41.654380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:41.655079 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:41.655095 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:41.655262 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:41.655868 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:41.655887 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:41.658575 (+ 2688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":272,"threads_started":1}
W20260810 14:32:41.659053 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:41.650853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:34443 (local address 127.6.46.254:35109)
0810 14:32:41.651026 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:41.651033 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:41.651427 (+ 394us) server_negotiation.cc:407] Connection header received
0810 14:32:41.651597 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:41.651601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:41.651663 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:41.651742 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:41.652544 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:41.653269 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:41.654506 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:41.654939 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:41.655049 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:41.658825 (+ 3776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:34443: BlockingRecv error: recv got EOF from 127.6.46.195:34443 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 14:32:41 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:41 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:41.780303 21816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:41.772304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:40545)
0810 14:32:41.772843 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:41.772858 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:41.772986 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:41.773304 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:41.773309 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:41.773723 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:41.773886 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:41.773896 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:41.774916 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:41.774921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:41.775844 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:41.775859 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:41.776633 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:41.776640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:41.776788 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:41.777304 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:41.777320 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:41.779969 (+ 2649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":284,"threads_started":1}
W20260810 14:32:41.780417 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:41.772405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:40545 (local address 127.6.46.254:35109)
0810 14:32:41.772605 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:41.772611 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:41.772940 (+ 329us) server_negotiation.cc:407] Connection header received
0810 14:32:41.773112 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:41.773115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:41.773176 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:41.773255 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:41.774052 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:41.774782 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:41.775988 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:41.776451 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:41.776560 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:41.780177 (+ 3617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:40545: BlockingRecv error: recv got EOF from 127.6.46.193:40545 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 14:32:42 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:42 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:42.214794 21817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:42.206634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:40247)
0810 14:32:42.207140 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:42.207155 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:42.207269 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:42.207523 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:42.207525 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:42.207892 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:42.208049 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:42.208058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:42.208975 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:42.208979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:42.210133 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:42.210149 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:42.211066 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:42.211073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:42.211253 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:42.211819 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:42.211838 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:42.214469 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":236,"threads_started":1}
W20260810 14:32:42.214924 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:42.206769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:40247 (local address 127.6.46.254:35109)
0810 14:32:42.206963 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:42.206969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:42.207225 (+ 256us) server_negotiation.cc:407] Connection header received
0810 14:32:42.207357 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:42.207360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:42.207420 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:42.207495 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:42.208160 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:42.208876 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:42.210292 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:42.210887 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:42.211031 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:42.214687 (+ 3656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:40247: BlockingRecv error: recv got EOF from 127.6.46.194:40247 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 14:32:42 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:42 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:42.667259 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:42.659786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:51673)
0810 14:32:42.660285 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:42.660301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:42.660411 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:42.660652 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:42.660655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:42.661044 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:42.661216 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:42.661225 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:42.662216 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:42.662220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:42.663120 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:42.663131 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:42.663776 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:42.663781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:42.663896 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:42.664408 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:42.664426 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:42.667022 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":236,"threads_started":1}
W20260810 14:32:42.667423 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:42.659840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:51673 (local address 127.6.46.254:35109)
0810 14:32:42.660001 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:42.660006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:42.660364 (+ 358us) server_negotiation.cc:407] Connection header received
0810 14:32:42.660496 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:42.660499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:42.660559 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:42.660636 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:42.661345 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:42.662115 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:42.663238 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:42.663641 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:42.663730 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:42.667235 (+ 3505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:51673: BlockingRecv error: recv got EOF from 127.6.46.195:51673 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 14:32:42 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:42 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:42.789441 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:42.781238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:53533)
0810 14:32:42.781667 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:42.781693 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:42.781802 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:42.782145 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:42.782148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:42.782499 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:42.782721 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:42.782734 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:42.783872 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:42.783876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:42.784783 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:42.784794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:42.785639 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:42.785644 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:42.785834 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:42.786433 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:42.786449 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:42.789090 (+ 2641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":221,"threads_started":1}
W20260810 14:32:42.789541 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:42.781321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:53533 (local address 127.6.46.254:35109)
0810 14:32:42.781583 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:42.781587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:42.781758 (+ 171us) server_negotiation.cc:407] Connection header received
0810 14:32:42.781916 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:42.781919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:42.782008 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:42.782096 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:42.782884 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:42.783746 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:42.784911 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:42.785488 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:42.785621 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:42.789298 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:53533: BlockingRecv error: recv got EOF from 127.6.46.193:53533 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
Aug 10 14:32:43 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:43 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:43.223387 21822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:43.215695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:55315)
0810 14:32:43.216108 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:43.216124 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:43.216239 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:43.216492 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:43.216495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:43.216866 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:43.217045 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:43.217056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:43.217979 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:43.217982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:43.218888 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:43.218900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:43.219544 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:43.219548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:43.219670 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:43.220257 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:43.220275 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:43.223035 (+ 2760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":207,"threads_started":1}
W20260810 14:32:43.223440 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:43.215781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:55315 (local address 127.6.46.254:35109)
0810 14:32:43.215983 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:43.215989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:43.216192 (+ 203us) server_negotiation.cc:407] Connection header received
0810 14:32:43.216319 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:43.216323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:43.216382 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:43.216456 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:43.217163 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:43.217877 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:43.219001 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:43.219417 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:43.219504 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:43.223234 (+ 3730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:55315: BlockingRecv error: recv got EOF from 127.6.46.194:55315 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 14:32:43 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:43 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.195@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:43.675849 21824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:43.668105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.195:46915)
0810 14:32:43.668532 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:43.668548 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:43.668659 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:43.668934 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:43.668937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:43.669311 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:43.669488 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:43.669496 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:43.670427 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:43.670430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:43.671330 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:43.671341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:43.672008 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:43.672014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:43.672194 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:43.672802 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:43.672826 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:43.675599 (+ 2773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":210,"threads_started":1}
W20260810 14:32:43.676023 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:43.668179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:46915 (local address 127.6.46.254:35109)
0810 14:32:43.668394 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:43.668400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:43.668622 (+ 222us) server_negotiation.cc:407] Connection header received
0810 14:32:43.668765 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:43.668769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:43.668829 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:43.668906 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:43.669592 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:43.670321 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:43.671433 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:43.671867 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:43.671973 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:43.675812 (+ 3839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.195:46915: BlockingRecv error: recv got EOF from 127.6.46.195:46915 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 14:32:43 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:43 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.193@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:43.798105 21825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:43.790571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.193:49363)
0810 14:32:43.791008 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:43.791024 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:43.791134 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:43.791428 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:43.791431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:43.791825 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:43.792017 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:43.792026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:43.792990 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:43.792993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:43.793890 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:43.793900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:43.794587 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:43.794591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:43.794728 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:43.795268 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:43.795284 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:43.797793 (+ 2509us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":235,"threads_started":1}
W20260810 14:32:43.798305 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:43.790623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:49363 (local address 127.6.46.254:35109)
0810 14:32:43.790837 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:43.790844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:43.791097 (+ 253us) server_negotiation.cc:407] Connection header received
0810 14:32:43.791236 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:43.791240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:43.791317 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:43.791395 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:43.792142 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:43.792886 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:43.793998 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:43.794463 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:43.794557 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:43.798063 (+ 3506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.193:49363: BlockingRecv error: recv got EOF from 127.6.46.193:49363 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 14:32:44 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 14:32:44 dist-test-slave-q9h9 krb5kdc[20080](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.46.194@KRBTEST.COM for oryx/127.6.46.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 14:32:44.232789 21826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:44.224510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:35109 (local address 127.6.46.194:37741)
0810 14:32:44.225019 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:44.225036 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:44.225155 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:44.225482 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:44.225486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:44.226045 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:44.226314 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:44.226328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:44.227484 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:44.227488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:44.228402 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:44.228415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:44.229092 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:44.229096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:44.229210 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:44.229744 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:44.229761 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:44.232543 (+ 2782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.46.254:35109: Server oryx/127.6.46.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":286,"threads_started":1}
W20260810 14:32:44.232928 21302 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:44.224607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:37741 (local address 127.6.46.254:35109)
0810 14:32:44.224814 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:44.224822 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:44.225115 (+ 293us) server_negotiation.cc:407] Connection header received
0810 14:32:44.225248 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:44.225251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:44.225313 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:44.225392 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:44.226468 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:44.227346 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:44.228541 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:44.228974 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:44.229075 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:44.232717 (+ 3642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.46.194:37741: BlockingRecv error: recv got EOF from 127.6.46.194:37741 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:32:44.513398 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 21057
I20260810 14:32:44.524503 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:44.514288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35300 (local address 127.6.46.193:34139)
0810 14:32:44.514784 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:44.514791 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:44.514837 (+ 46us) server_negotiation.cc:407] Connection header received
0810 14:32:44.514892 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:44.514899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:44.515106 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:44.515255 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:44.516545 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:44.517490 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:44.518667 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:44.519212 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:44.519308 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:44.521539 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:44.521547 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:44.521561 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:44.521594 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:44.524050 (+ 2456us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.46.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 14:32:44.524286 (+ 236us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35300: Request ticket server kudu/127.6.46.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":223,"threads_started":1}
W20260810 14:32:44.524602 21829 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35300: Request ticket server kudu/127.6.46.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 14:32:44.524672 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 21057: Not authorized: Client connection negotiation failed: client connection to 127.6.46.193:34139: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.46.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 14:32:44.524860 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 21057
I20260810 14:32:44.549383 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 21303
I20260810 14:32:44.560318 21830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:44.550414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.6.46.194:45229)
0810 14:32:44.551017 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:44.551024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:44.551053 (+ 29us) server_negotiation.cc:407] Connection header received
0810 14:32:44.551111 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:44.551118 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:44.551322 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:44.551488 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:44.552593 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:44.553615 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:44.554859 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:44.555392 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:44.555483 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:44.557640 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:44.557655 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:44.557667 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:44.557700 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:44.559865 (+ 2165us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.46.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 14:32:44.560085 (+ 220us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36318: Request ticket server kudu/127.6.46.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":296,"threads_started":1}
W20260810 14:32:44.560405 21830 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36318: Request ticket server kudu/127.6.46.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 14:32:44.560446 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 21303: Not authorized: Client connection negotiation failed: client connection to 127.6.46.194:45229: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.46.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 14:32:44.560595 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 21303
I20260810 14:32:44.588326 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 21526
I20260810 14:32:44.599133 21831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:44.589099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45164 (local address 127.6.46.195:45057)
0810 14:32:44.589698 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:44.589705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:44.589732 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:32:44.589792 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:44.589798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:44.590009 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:44.590205 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:44.591372 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:44.592393 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:44.593427 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:44.593965 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:44.594176 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:44.596410 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:44.596417 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:44.596430 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:44.596458 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:44.598663 (+ 2205us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.46.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 14:32:44.598903 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45164: Request ticket server kudu/127.6.46.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":318,"threads_started":1}
W20260810 14:32:44.599225 21831 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45164: Request ticket server kudu/127.6.46.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 14:32:44.599272 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 21526: Not authorized: Client connection negotiation failed: client connection to 127.6.46.195:45057: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.46.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 14:32:44.599411 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 21526
I20260810 14:32:44.621886 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 20937
I20260810 14:32:44.623109 21050 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:44.736105 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 20937
2026-08-10T14:32:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21063 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T14:32:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:32:44Z Disabled control of system clock
I20260810 14:32:44.820621 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:46753
--webserver_interface=127.6.46.254
--webserver_port=0
--builtin_ntp_servers=127.6.46.212:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:46753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 14:32:45.146194 21839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:45.146445 21839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:45.146529 21839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:45.153357 21839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:32:45.153443 21839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:45.153483 21839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:45.153540 21839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:32:45.153584 21839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:32:45.162144 21839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:46753
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:46753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:45.163246 21839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:45.164839 21839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:45.175777 21850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:45.175843 21848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:45.175833 21852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:45.176652 21839 server_base.cc:1034] running on GCE node
I20260810 14:32:45.177292 21839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:45.178529 21839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:45.179723 21839 hybrid_clock.cc:648] HybridClock initialized: now 1786372365179671 us; error 45 us; skew 500 ppm
I20260810 14:32:45.180037 21839 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:32:45.182132 21839 webserver.cc:460] Webserver started at http://127.6.46.254:38373/ using document root <none> and password file <none>
I20260810 14:32:45.182758 21839 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:45.182848 21839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:45.183158 21839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:45.185784 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "18e93ba7ec334b9dab20dee6e8041723"
format_stamp: "Formatted at 2026-08-10 14:32:45 on dist-test-slave-q9h9"
server_key: "97463db643fb4409db3c9cb5acb364c4"
server_key_iv: "9e46a662ece25858ec902e743f690982"
server_key_version: "encryptionkey@0"
I20260810 14:32:45.186707 21839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "18e93ba7ec334b9dab20dee6e8041723"
format_stamp: "Formatted at 2026-08-10 14:32:45 on dist-test-slave-q9h9"
server_key: "97463db643fb4409db3c9cb5acb364c4"
server_key_iv: "9e46a662ece25858ec902e743f690982"
server_key_version: "encryptionkey@0"
I20260810 14:32:45.191304 21839 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 14:32:45.194463 21862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:45.195648 21839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260810 14:32:45.195801 21839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "18e93ba7ec334b9dab20dee6e8041723"
format_stamp: "Formatted at 2026-08-10 14:32:45 on dist-test-slave-q9h9"
server_key: "97463db643fb4409db3c9cb5acb364c4"
server_key_iv: "9e46a662ece25858ec902e743f690982"
server_key_version: "encryptionkey@0"
I20260810 14:32:45.195983 21839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:45.211483 21839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:45.212167 21839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:45.212409 21839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:45.228444 21839 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:46753
I20260810 14:32:45.228456 21946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:46753 every 8 connection(s)
I20260810 14:32:45.229585 21839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:32:45.232892 21948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:45.238580 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 21839
I20260810 14:32:45.238811 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:32:45.239624 6331 external_mini_cluster.cc:1428] Setting key bd6c179c69d16e23f116b69f86994eee
I20260810 14:32:45.243402 21948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723: Bootstrap starting.
I20260810 14:32:45.246279 21948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:45.247203 21948 log.cc:824] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:45.250423 21948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723: No bootstrap required, opened a new log
I20260810 14:32:45.253607 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:45.241586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48868 (local address 127.6.46.254:46753)
0810 14:32:45.242513 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:45.242523 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:45.242561 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:32:45.244187 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:45.244208 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:45.244624 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:45.244950 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:45.245517 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:45.246797 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:45.248499 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:45.249522 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:45.249740 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:45.250794 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:45.250804 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:45.250826 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:45.251522 (+ 696us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:45.251644 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:45.251672 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:45.251783 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:45.251999 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 14:32:45.252337 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":330,"threads_started":1}
I20260810 14:32:45.258682 21948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e93ba7ec334b9dab20dee6e8041723" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46753 } }
I20260810 14:32:45.258982 21948 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:45.259071 21948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18e93ba7ec334b9dab20dee6e8041723, State: Initialized, Role: FOLLOWER
I20260810 14:32:45.259523 21948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e93ba7ec334b9dab20dee6e8041723" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46753 } }
I20260810 14:32:45.259706 21948 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:32:45.259796 21948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:32:45.259913 21948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:45.261920 21948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e93ba7ec334b9dab20dee6e8041723" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46753 } }
I20260810 14:32:45.262449 21948 leader_election.cc:302] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18e93ba7ec334b9dab20dee6e8041723; no voters:
I20260810 14:32:45.262965 21948 leader_election.cc:288] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:32:45.263094 21955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:45.263705 21955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [term 1 LEADER]: Becoming Leader. State: Replica: 18e93ba7ec334b9dab20dee6e8041723, State: Running, Role: LEADER
I20260810 14:32:45.264197 21955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e93ba7ec334b9dab20dee6e8041723" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46753 } }
I20260810 14:32:45.264835 21948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:32:45.266826 21960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18e93ba7ec334b9dab20dee6e8041723. Latest consensus state: current_term: 1 leader_uuid: "18e93ba7ec334b9dab20dee6e8041723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e93ba7ec334b9dab20dee6e8041723" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46753 } } }
I20260810 14:32:45.267225 21960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:45.267702 21956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18e93ba7ec334b9dab20dee6e8041723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18e93ba7ec334b9dab20dee6e8041723" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46753 } } }
I20260810 14:32:45.267997 21956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:45.268834 21966 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:32:45.274206 21966 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:32:45.282060 21966 catalog_manager.cc:1347] Generated new cluster ID: 919084af2e27462aba453cc6420c661d
I20260810 14:32:45.282205 21966 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:32:45.293980 21966 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:32:45.295218 21966 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:32:45.304055 21966 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18e93ba7ec334b9dab20dee6e8041723: Generated new TSK 0
I20260810 14:32:45.304714 21966 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:32:45.316425 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46753
--builtin_ntp_servers=127.6.46.212:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 14:32:45.635502 21990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:45.635767 21990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:45.635877 21990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:45.643076 21990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:45.643163 21990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:45.643304 21990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:32:45.652042 21990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:45.653205 21990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:45.654943 21990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:45.666896 22000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:45.666972 21996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:45.666935 21995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:45.667614 21990 server_base.cc:1034] running on GCE node
I20260810 14:32:45.668156 21990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:45.668957 21990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:45.670128 21990 hybrid_clock.cc:648] HybridClock initialized: now 1786372365670083 us; error 32 us; skew 500 ppm
I20260810 14:32:45.670414 21990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:32:45.676723 21990 webserver.cc:460] Webserver started at http://127.6.46.193:41387/ using document root <none> and password file <none>
I20260810 14:32:45.677362 21990 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:45.677460 21990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:45.677758 21990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:45.680423 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "48a3c5c1a41a4f7fa9901a0ce4dec5dd"
format_stamp: "Formatted at 2026-08-10 14:32:45 on dist-test-slave-q9h9"
server_key: "07d21bb31b89e381df131194bc01b08a"
server_key_iv: "0ea9e2b023502ed06e6929639f13fe5b"
server_key_version: "encryptionkey@0"
I20260810 14:32:45.681175 21990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "48a3c5c1a41a4f7fa9901a0ce4dec5dd"
format_stamp: "Formatted at 2026-08-10 14:32:45 on dist-test-slave-q9h9"
server_key: "07d21bb31b89e381df131194bc01b08a"
server_key_iv: "0ea9e2b023502ed06e6929639f13fe5b"
server_key_version: "encryptionkey@0"
I20260810 14:32:45.685804 21990 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:32:45.688977 22012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:45.690214 21990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:32:45.690352 21990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "48a3c5c1a41a4f7fa9901a0ce4dec5dd"
format_stamp: "Formatted at 2026-08-10 14:32:45 on dist-test-slave-q9h9"
server_key: "07d21bb31b89e381df131194bc01b08a"
server_key_iv: "0ea9e2b023502ed06e6929639f13fe5b"
server_key_version: "encryptionkey@0"
I20260810 14:32:45.690531 21990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:45.725824 21990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:45.726603 21990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:45.726871 21990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:45.727741 21990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:45.729065 21990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:45.729159 21990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:45.729260 21990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:45.729331 21990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:45.758904 21990 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:35267
I20260810 14:32:45.758965 22221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:35267 every 8 connection(s)
I20260810 14:32:45.760084 21990 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:32:45.762344 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 21990
I20260810 14:32:45.762569 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:32:45.763355 6331 external_mini_cluster.cc:1428] Setting key 2df8319931a3c9abf5393bbe962b9aa0
I20260810 14:32:45.766448 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46753
--builtin_ntp_servers=127.6.46.212:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 14:32:45.773242 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:45.762488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:58505 (local address 127.6.46.254:46753)
0810 14:32:45.765215 (+ 2727us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:45.765221 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:45.765242 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:32:45.766807 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:45.766813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:45.766900 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:45.766992 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:45.768003 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:45.769005 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:45.770624 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:45.771010 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:45.771124 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:45.772408 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:45.772414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:45.772419 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:45.772436 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:45.772476 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:45.772487 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:45.772561 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:45.772785 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 14:32:45.772883 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2608,"thread_start_us":2473,"threads_started":1}
I20260810 14:32:45.774196 22226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:45.763162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:46753 (local address 127.6.46.193:58505)
0810 14:32:45.764755 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:45.764798 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:45.766528 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:45.767226 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:45.767234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:45.767317 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:45.767845 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:45.767863 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:45.769145 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:45.769149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:45.770470 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:45.770483 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:45.771222 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:45.771231 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:45.771472 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:45.772102 (+ 630us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:32:45.772105 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:45.772153 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:32:45.772232 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:45.772581 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:45.772598 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:45.772604 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:32:45.772750 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 14:32:45.773150 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"thread_start_us":874,"threads_started":1}
I20260810 14:32:45.775588 22222 heartbeater.cc:347] Connected to a master server at 127.6.46.254:46753
I20260810 14:32:45.775897 22222 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:45.776739 22222 heartbeater.cc:511] Master 127.6.46.254:46753 requested a full tablet report, sending...
I20260810 14:32:45.778988 21889 ts_manager.cc:194] Registered new tserver with Master: 48a3c5c1a41a4f7fa9901a0ce4dec5dd (127.6.46.193:35267)
I20260810 14:32:45.780891 21889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.46.193:58505
W20260810 14:32:46.081199 22228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:46.081449 22228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:46.081529 22228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:46.088388 22228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:46.088474 22228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:46.088577 22228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:32:46.097231 22228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:46.098495 22228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:46.100255 22228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:46.112390 22236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:46.112385 22247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:46.112634 22235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:46.113111 22228 server_base.cc:1034] running on GCE node
I20260810 14:32:46.113647 22228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:46.114583 22228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:46.115787 22228 hybrid_clock.cc:648] HybridClock initialized: now 1786372366115738 us; error 57 us; skew 500 ppm
I20260810 14:32:46.116073 22228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:32:46.118152 22228 webserver.cc:460] Webserver started at http://127.6.46.194:44833/ using document root <none> and password file <none>
I20260810 14:32:46.118770 22228 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:46.118865 22228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:46.119175 22228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:46.121773 22228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "af0f150b471c4b42a99be95172893d76"
format_stamp: "Formatted at 2026-08-10 14:32:46 on dist-test-slave-q9h9"
server_key: "69f3580dbf250e9050fd6cbee2d399f7"
server_key_iv: "5a676e916c9fffba111a872682d3ad52"
server_key_version: "encryptionkey@0"
I20260810 14:32:46.122606 22228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "af0f150b471c4b42a99be95172893d76"
format_stamp: "Formatted at 2026-08-10 14:32:46 on dist-test-slave-q9h9"
server_key: "69f3580dbf250e9050fd6cbee2d399f7"
server_key_iv: "5a676e916c9fffba111a872682d3ad52"
server_key_version: "encryptionkey@0"
I20260810 14:32:46.127405 22228 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:32:46.130595 22266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:46.131811 22228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:32:46.131963 22228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "af0f150b471c4b42a99be95172893d76"
format_stamp: "Formatted at 2026-08-10 14:32:46 on dist-test-slave-q9h9"
server_key: "69f3580dbf250e9050fd6cbee2d399f7"
server_key_iv: "5a676e916c9fffba111a872682d3ad52"
server_key_version: "encryptionkey@0"
I20260810 14:32:46.132192 22228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:46.156340 22228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:46.157117 22228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:46.157392 22228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:46.158303 22228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:46.159713 22228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:46.159809 22228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:46.159916 22228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:46.159986 22228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:46.189505 22228 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:46509
I20260810 14:32:46.189576 22463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:46509 every 8 connection(s)
I20260810 14:32:46.190757 22228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:32:46.193220 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 22228
I20260810 14:32:46.193408 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:32:46.193979 6331 external_mini_cluster.cc:1428] Setting key 43d97227950f24ba7ad74694c8f9b3dd
I20260810 14:32:46.196736 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46753
--builtin_ntp_servers=127.6.46.212:44907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 14:32:46.202828 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:46.193020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:47403 (local address 127.6.46.254:46753)
0810 14:32:46.193188 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:46.193209 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:46.195016 (+ 1807us) server_negotiation.cc:407] Connection header received
0810 14:32:46.196836 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:46.196842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:46.196919 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:46.197003 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:46.198005 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.198754 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.200317 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.200720 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.200830 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:46.202058 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:46.202065 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:46.202070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:46.202091 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:46.202123 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:46.202135 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:46.202210 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:46.202412 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 14:32:46.202506 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:32:46.203693 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:46.193704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:46753 (local address 127.6.46.194:47403)
0810 14:32:46.194802 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:46.194919 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:46.196592 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:46.197236 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:46.197244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:46.197328 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:46.197848 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:46.197864 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.198906 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.198911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:46.200187 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:46.200200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.200925 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.200932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:46.201149 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:46.201758 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:32:46.201761 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:46.201795 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:32:46.201870 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:46.202223 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:46.202234 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:46.202241 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:32:46.202388 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 14:32:46.202737 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":349,"threads_started":1}
I20260810 14:32:46.204963 22464 heartbeater.cc:347] Connected to a master server at 127.6.46.254:46753
I20260810 14:32:46.205255 22464 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:46.206089 22464 heartbeater.cc:511] Master 127.6.46.254:46753 requested a full tablet report, sending...
I20260810 14:32:46.207487 21889 ts_manager.cc:194] Registered new tserver with Master: af0f150b471c4b42a99be95172893d76 (127.6.46.194:46509)
I20260810 14:32:46.208257 21889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.46.194:47403
W20260810 14:32:46.513406 22468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:46.513653 22468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:46.513772 22468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:46.520884 22468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:46.520987 22468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:46.521124 22468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:32:46.529737 22468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:46753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:46.530943 22468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:46.532599 22468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:46.544757 22477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:46.544792 22479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:46.544816 22475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:46.545451 22468 server_base.cc:1034] running on GCE node
I20260810 14:32:46.546046 22468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:46.546922 22468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:46.548115 22468 hybrid_clock.cc:648] HybridClock initialized: now 1786372366548061 us; error 50 us; skew 500 ppm
I20260810 14:32:46.548396 22468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:32:46.550514 22468 webserver.cc:460] Webserver started at http://127.6.46.195:40387/ using document root <none> and password file <none>
I20260810 14:32:46.551136 22468 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:46.551236 22468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:46.551532 22468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:46.554121 22468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "5f925ee3e73842d18da3114d7e4e01a9"
format_stamp: "Formatted at 2026-08-10 14:32:46 on dist-test-slave-q9h9"
server_key: "23bd59a4159a7a8b52a7a0c9c3979a9e"
server_key_iv: "fc295b2edb61849f0df642fcad557e0c"
server_key_version: "encryptionkey@0"
I20260810 14:32:46.554867 22468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "5f925ee3e73842d18da3114d7e4e01a9"
format_stamp: "Formatted at 2026-08-10 14:32:46 on dist-test-slave-q9h9"
server_key: "23bd59a4159a7a8b52a7a0c9c3979a9e"
server_key_iv: "fc295b2edb61849f0df642fcad557e0c"
server_key_version: "encryptionkey@0"
I20260810 14:32:46.559492 22468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:32:46.562565 22486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:46.563697 22468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:32:46.563843 22468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "5f925ee3e73842d18da3114d7e4e01a9"
format_stamp: "Formatted at 2026-08-10 14:32:46 on dist-test-slave-q9h9"
server_key: "23bd59a4159a7a8b52a7a0c9c3979a9e"
server_key_iv: "fc295b2edb61849f0df642fcad557e0c"
server_key_version: "encryptionkey@0"
I20260810 14:32:46.564024 22468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:46.578109 22468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:46.578796 22468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:46.579063 22468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:46.579870 22468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:46.581151 22468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:46.581243 22468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:46.581346 22468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:46.581418 22468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:46.611340 22468 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:34417
I20260810 14:32:46.611402 22666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:34417 every 8 connection(s)
I20260810 14:32:46.612574 22468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:32:46.621905 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 22468
I20260810 14:32:46.622117 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:32:46.622509 6331 external_mini_cluster.cc:1428] Setting key 0997738e3fb050a1788d8ae3e9bdb0b4
I20260810 14:32:46.624224 22225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:46.615027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:60505 (local address 127.6.46.254:46753)
0810 14:32:46.615236 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:46.615241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:46.616468 (+ 1227us) server_negotiation.cc:407] Connection header received
0810 14:32:46.618147 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:46.618152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:46.618222 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:46.618310 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:46.619265 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.619983 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.621465 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.621873 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.622009 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:46.623400 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:46.623405 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:46.623408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:46.623427 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:46.623460 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:46.623472 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:46.623549 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:46.623783 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 14:32:46.623885 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 14:32:46.625067 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:46.615482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:46753 (local address 127.6.46.195:60505)
0810 14:32:46.616322 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:46.616364 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:46.617868 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:46.618483 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:46.618491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:46.618580 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:46.619111 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:46.619127 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.620108 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.620112 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:46.621307 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:46.621319 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.622177 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.622187 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:46.622384 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:46.623076 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:32:46.623080 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:46.623130 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:32:46.623218 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:46.623571 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:46.623582 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:46.623589 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:32:46.623762 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 14:32:46.624043 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":255,"threads_started":1}
I20260810 14:32:46.626250 22667 heartbeater.cc:347] Connected to a master server at 127.6.46.254:46753
I20260810 14:32:46.626499 22667 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:46.627233 22667 heartbeater.cc:511] Master 127.6.46.254:46753 requested a full tablet report, sending...
I20260810 14:32:46.628458 21889 ts_manager.cc:194] Registered new tserver with Master: 5f925ee3e73842d18da3114d7e4e01a9 (127.6.46.195:34417)
I20260810 14:32:46.629210 21889 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.46.195:60505
I20260810 14:32:46.637880 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 14:32:46.643812 22225 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:46.642385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48876 (local address 127.6.46.254:46753)
0810 14:32:46.642618 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:46.642625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:46.642972 (+ 347us) server_negotiation.cc:407] Connection header received
0810 14:32:46.643106 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:46.643109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:46.643182 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:46.643259 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:46.643591 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48876: BlockingRecv error: recv got EOF from 127.0.0.1:48876 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:32:46.644469 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 21990
I20260810 14:32:46.651513 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:46.645374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.6.46.193:35267)
0810 14:32:46.645863 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:46.645869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:46.645894 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:32:46.645946 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:46.645953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:46.646187 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:46.646310 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:46.646629 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.647609 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.648778 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.649288 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.649379 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:46.650265 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:46.650273 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:46.650284 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:46.650852 (+ 568us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:46.650899 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:46.650919 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:46.651003 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:46.651125 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 14:32:46.651231 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":242,"threads_started":1}
I20260810 14:32:46.652633 22215 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:46.731254 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 21990
I20260810 14:32:46.752511 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 22228
I20260810 14:32:46.759635 22685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:46.753376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49362 (local address 127.6.46.194:46509)
0810 14:32:46.753901 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:46.753908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:46.753934 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:32:46.753994 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:46.754000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:46.754247 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:46.754365 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:46.754699 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.755670 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.756794 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.757343 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.757438 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:46.758326 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:46.758335 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:46.758347 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:46.758917 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:46.758964 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:46.758983 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:46.759072 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:46.759159 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 14:32:46.759265 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":267,"threads_started":1}
I20260810 14:32:46.760843 22457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:46.839823 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 22228
I20260810 14:32:46.859076 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 22468
I20260810 14:32:46.867125 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:46.859928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.6.46.195:34417)
0810 14:32:46.860532 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:46.860538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:46.860572 (+ 34us) server_negotiation.cc:407] Connection header received
0810 14:32:46.860634 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:46.860643 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:46.860859 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:46.861001 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:46.861450 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.862550 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.863808 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:46.864369 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:46.864472 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:46.865573 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:46.865584 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:46.865598 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:46.866382 (+ 784us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:46.866429 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:46.866451 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:46.866569 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:46.866713 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 14:32:46.866824 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":293,"threads_started":1}
I20260810 14:32:46.868428 22661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:46.947871 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 22468
I20260810 14:32:46.971966 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 21839
I20260810 14:32:46.973165 21942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:47.073886 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 21839
2026-08-10T14:32:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2332 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T14:32:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:32:47Z Disabled control of system clock
I20260810 14:32:47.155364 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:38117
--webserver_interface=127.6.46.254
--webserver_port=0
--builtin_ntp_servers=127.6.46.212:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:38117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 14:32:47.472476 22696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:47.472746 22696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:47.472831 22696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:47.479849 22696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:32:47.479950 22696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:47.480000 22696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:47.480058 22696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:32:47.480101 22696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:32:47.488899 22696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:38117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:38117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:47.490152 22696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:47.491978 22696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:47.503175 22705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:47.503206 22706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:47.503297 22709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:47.503958 22696 server_base.cc:1034] running on GCE node
I20260810 14:32:47.504657 22696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:47.505951 22696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:47.507156 22696 hybrid_clock.cc:648] HybridClock initialized: now 1786372367507110 us; error 42 us; skew 500 ppm
I20260810 14:32:47.507444 22696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:32:47.509469 22696 webserver.cc:460] Webserver started at http://127.6.46.254:36049/ using document root <none> and password file <none>
I20260810 14:32:47.510154 22696 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:47.510246 22696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:47.510522 22696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:47.513218 22696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "c8ef255b85854b1e914e450ae6ff64d2"
format_stamp: "Formatted at 2026-08-10 14:32:47 on dist-test-slave-q9h9"
server_key: "d0b22473ff53184241e8961a2b2dbf35"
server_key_iv: "4a5cb464265806c7f083e6acc7725945"
server_key_version: "encryptionkey@0"
I20260810 14:32:47.513955 22696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "c8ef255b85854b1e914e450ae6ff64d2"
format_stamp: "Formatted at 2026-08-10 14:32:47 on dist-test-slave-q9h9"
server_key: "d0b22473ff53184241e8961a2b2dbf35"
server_key_iv: "4a5cb464265806c7f083e6acc7725945"
server_key_version: "encryptionkey@0"
I20260810 14:32:47.518919 22696 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 14:32:47.522315 22717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:47.523547 22696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:32:47.523692 22696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "c8ef255b85854b1e914e450ae6ff64d2"
format_stamp: "Formatted at 2026-08-10 14:32:47 on dist-test-slave-q9h9"
server_key: "d0b22473ff53184241e8961a2b2dbf35"
server_key_iv: "4a5cb464265806c7f083e6acc7725945"
server_key_version: "encryptionkey@0"
I20260810 14:32:47.523890 22696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:47.529246 22696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:47.529954 22696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:47.530282 22696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:47.546764 22696 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:38117
I20260810 14:32:47.546797 22815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:38117 every 8 connection(s)
I20260810 14:32:47.547971 22696 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:32:47.551499 22816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:47.555102 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 22696
I20260810 14:32:47.555279 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:32:47.555927 6331 external_mini_cluster.cc:1428] Setting key fa980e59d57932686bc2bc300107951f
I20260810 14:32:47.561914 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2: Bootstrap starting.
I20260810 14:32:47.563794 22825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:47.557735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.6.46.254:38117)
0810 14:32:47.558672 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:47.558682 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:47.558723 (+ 41us) server_negotiation.cc:407] Connection header received
0810 14:32:47.560502 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:47.560524 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:47.560892 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:47.561195 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:47.562322 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:47.562328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:47.562340 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:47.562356 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:47.562419 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:47.562435 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:47.562604 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:47.562697 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 14:32:47.562934 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":364,"threads_started":1}
I20260810 14:32:47.565277 22816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:47.566314 22816 log.cc:824] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:47.568758 22816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2: No bootstrap required, opened a new log
I20260810 14:32:47.575726 22816 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ef255b85854b1e914e450ae6ff64d2" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38117 } }
I20260810 14:32:47.576020 22816 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:47.576110 22816 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8ef255b85854b1e914e450ae6ff64d2, State: Initialized, Role: FOLLOWER
I20260810 14:32:47.576637 22816 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ef255b85854b1e914e450ae6ff64d2" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38117 } }
I20260810 14:32:47.576841 22816 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:32:47.576964 22816 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:32:47.577106 22816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:47.579119 22816 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ef255b85854b1e914e450ae6ff64d2" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38117 } }
I20260810 14:32:47.579551 22816 leader_election.cc:302] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8ef255b85854b1e914e450ae6ff64d2; no voters:
I20260810 14:32:47.580129 22816 leader_election.cc:288] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:32:47.580228 22828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:47.580964 22828 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [term 1 LEADER]: Becoming Leader. State: Replica: c8ef255b85854b1e914e450ae6ff64d2, State: Running, Role: LEADER
I20260810 14:32:47.581459 22828 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ef255b85854b1e914e450ae6ff64d2" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38117 } }
I20260810 14:32:47.582007 22816 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:32:47.584151 22830 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8ef255b85854b1e914e450ae6ff64d2. Latest consensus state: current_term: 1 leader_uuid: "c8ef255b85854b1e914e450ae6ff64d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ef255b85854b1e914e450ae6ff64d2" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38117 } } }
I20260810 14:32:47.584215 22829 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8ef255b85854b1e914e450ae6ff64d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8ef255b85854b1e914e450ae6ff64d2" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38117 } } }
I20260810 14:32:47.584515 22830 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:47.584579 22829 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:47.586227 22837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:32:47.590710 22837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:32:47.600036 22837 catalog_manager.cc:1347] Generated new cluster ID: 04798ced112941a6948acb07625950e6
I20260810 14:32:47.600142 22837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:32:47.617066 22837 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:32:47.617430 22837 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:32:47.629257 22837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8ef255b85854b1e914e450ae6ff64d2: Generated new TSK 0
I20260810 14:32:47.630232 22837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:32:47.647140 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38117
--builtin_ntp_servers=127.6.46.212:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 14:32:47.972159 22855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:47.972406 22855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:47.972520 22855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:47.979465 22855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:47.979578 22855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:47.979733 22855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:32:47.988518 22855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:47.989696 22855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:47.991945 22855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:48.006938 22866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:48.006978 22869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:48.006899 22864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:48.013511 22855 server_base.cc:1034] running on GCE node
I20260810 14:32:48.014133 22855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:48.015017 22855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:48.016217 22855 hybrid_clock.cc:648] HybridClock initialized: now 1786372368016160 us; error 46 us; skew 500 ppm
I20260810 14:32:48.016511 22855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:32:48.018675 22855 webserver.cc:460] Webserver started at http://127.6.46.193:38419/ using document root <none> and password file <none>
I20260810 14:32:48.019305 22855 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:48.019402 22855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:48.019723 22855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:48.022475 22855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "f5221de1e5b7423e858789eb3bdd9d2c"
format_stamp: "Formatted at 2026-08-10 14:32:48 on dist-test-slave-q9h9"
server_key: "9c55e1f7979244cdeb5de51ba0e2c445"
server_key_iv: "0a034576e7bba6442ad364a157fed421"
server_key_version: "encryptionkey@0"
I20260810 14:32:48.023221 22855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "f5221de1e5b7423e858789eb3bdd9d2c"
format_stamp: "Formatted at 2026-08-10 14:32:48 on dist-test-slave-q9h9"
server_key: "9c55e1f7979244cdeb5de51ba0e2c445"
server_key_iv: "0a034576e7bba6442ad364a157fed421"
server_key_version: "encryptionkey@0"
I20260810 14:32:48.028085 22855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 14:32:48.031350 22882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:48.032546 22855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:32:48.032701 22855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "f5221de1e5b7423e858789eb3bdd9d2c"
format_stamp: "Formatted at 2026-08-10 14:32:48 on dist-test-slave-q9h9"
server_key: "9c55e1f7979244cdeb5de51ba0e2c445"
server_key_iv: "0a034576e7bba6442ad364a157fed421"
server_key_version: "encryptionkey@0"
I20260810 14:32:48.032917 22855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:48.038970 22855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:48.039669 22855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:48.039899 22855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:48.040726 22855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:48.041962 22855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:48.042083 22855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:48.042186 22855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:48.042253 22855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:48.070781 22855 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:38173
I20260810 14:32:48.070820 23078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:38173 every 8 connection(s)
I20260810 14:32:48.072510 22855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:32:48.073628 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 22855
I20260810 14:32:48.073796 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:32:48.074544 6331 external_mini_cluster.cc:1428] Setting key b67fcbddbdb86ee7c177cf318ac8ee6f
I20260810 14:32:48.077441 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38117
--builtin_ntp_servers=127.6.46.212:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 14:32:48.080649 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:48.075007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:53597 (local address 127.6.46.254:38117)
0810 14:32:48.075408 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:48.075413 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:48.076903 (+ 1490us) server_negotiation.cc:407] Connection header received
0810 14:32:48.078616 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:48.078622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:48.078694 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:48.078806 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:48.079979 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:48.079983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:48.079987 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:48.080004 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:48.080055 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:48.080069 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:48.080175 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:48.080323 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 14:32:48.080380 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":194,"threads_started":1}
I20260810 14:32:48.081413 23088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:48.075691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:38117 (local address 127.6.46.193:53597)
0810 14:32:48.076729 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:48.076788 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:48.078346 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:48.078994 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:48.079001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:48.079086 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:48.079704 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:32:48.079707 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:48.079745 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:32:48.079822 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:48.080153 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:48.080159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:48.080164 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:32:48.080366 (+ 202us) client_negotiation.cc:240] Negotiation successful
0810 14:32:48.080545 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":316,"threads_started":1}
I20260810 14:32:48.082821 23080 heartbeater.cc:347] Connected to a master server at 127.6.46.254:38117
I20260810 14:32:48.083103 23080 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:48.083753 23080 heartbeater.cc:511] Master 127.6.46.254:38117 requested a full tablet report, sending...
I20260810 14:32:48.085987 22740 ts_manager.cc:194] Registered new tserver with Master: f5221de1e5b7423e858789eb3bdd9d2c (127.6.46.193:38173)
W20260810 14:32:48.400460 23089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:48.400709 23089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:48.400818 23089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:48.407641 23089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:48.407724 23089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:48.407855 23089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:32:48.416290 23089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:48.417439 23089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:48.419104 23089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:48.431277 23098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:48.431223 23101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:48.431720 23097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:48.432080 23089 server_base.cc:1034] running on GCE node
I20260810 14:32:48.432611 23089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:48.433431 23089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:48.434616 23089 hybrid_clock.cc:648] HybridClock initialized: now 1786372368434576 us; error 31 us; skew 500 ppm
I20260810 14:32:48.434911 23089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:32:48.436923 23089 webserver.cc:460] Webserver started at http://127.6.46.194:46437/ using document root <none> and password file <none>
I20260810 14:32:48.437582 23089 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:48.437680 23089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:48.437981 23089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:48.440608 23089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "d0e83651e81a4eeda56eb5feb8e7f099"
format_stamp: "Formatted at 2026-08-10 14:32:48 on dist-test-slave-q9h9"
server_key: "209693fc1b19b1edad0c845daf6ca39b"
server_key_iv: "6ee59c37a7afd38bedc95338f576bb97"
server_key_version: "encryptionkey@0"
I20260810 14:32:48.441327 23089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "d0e83651e81a4eeda56eb5feb8e7f099"
format_stamp: "Formatted at 2026-08-10 14:32:48 on dist-test-slave-q9h9"
server_key: "209693fc1b19b1edad0c845daf6ca39b"
server_key_iv: "6ee59c37a7afd38bedc95338f576bb97"
server_key_version: "encryptionkey@0"
I20260810 14:32:48.446038 23089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260810 14:32:48.449183 23110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:48.450388 23089 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260810 14:32:48.450531 23089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "d0e83651e81a4eeda56eb5feb8e7f099"
format_stamp: "Formatted at 2026-08-10 14:32:48 on dist-test-slave-q9h9"
server_key: "209693fc1b19b1edad0c845daf6ca39b"
server_key_iv: "6ee59c37a7afd38bedc95338f576bb97"
server_key_version: "encryptionkey@0"
I20260810 14:32:48.450718 23089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:48.456033 23089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:48.456720 23089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:48.456948 23089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:48.457768 23089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:48.459112 23089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:48.459208 23089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:48.459311 23089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:48.459380 23089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:48.488799 23089 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:45091
I20260810 14:32:48.488858 23305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:45091 every 8 connection(s)
I20260810 14:32:48.489965 23089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:32:48.493270 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 23089
I20260810 14:32:48.493422 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:32:48.493848 6331 external_mini_cluster.cc:1428] Setting key 0abcb9d631339bc78726ae77854689b1
I20260810 14:32:48.496714 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38117
--builtin_ntp_servers=127.6.46.212:45517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 14:32:48.497885 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:48.492377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:48163 (local address 127.6.46.254:38117)
0810 14:32:48.492603 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:48.492609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:48.494034 (+ 1425us) server_negotiation.cc:407] Connection header received
0810 14:32:48.495665 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:48.495670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:48.495738 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:48.495820 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:48.497201 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:48.497205 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:48.497208 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:48.497224 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:48.497266 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:48.497278 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:48.497391 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:48.497615 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 14:32:48.497665 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 14:32:48.498983 23310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:48.492844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:38117 (local address 127.6.46.194:48163)
0810 14:32:48.493855 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:48.493897 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:48.495420 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:48.496069 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:48.496079 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:48.496192 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:48.496873 (+ 681us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:32:48.496877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:48.496939 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:32:48.497040 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:48.497398 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:48.497409 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:48.497421 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 14:32:48.497690 (+ 269us) client_negotiation.cc:240] Negotiation successful
0810 14:32:48.497918 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":324,"threads_started":1}
I20260810 14:32:48.500200 23307 heartbeater.cc:347] Connected to a master server at 127.6.46.254:38117
I20260810 14:32:48.500468 23307 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:48.501001 23307 heartbeater.cc:511] Master 127.6.46.254:38117 requested a full tablet report, sending...
I20260810 14:32:48.502279 22740 ts_manager.cc:194] Registered new tserver with Master: d0e83651e81a4eeda56eb5feb8e7f099 (127.6.46.194:45091)
W20260810 14:32:48.819940 23311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:48.820216 23311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:48.820335 23311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:48.827373 23311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:48.827468 23311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:48.827633 23311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:32:48.836301 23311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38117
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:48.837495 23311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:48.839247 23311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:48.851241 23323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:48.851395 23326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:48.851730 23322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:48.852389 23311 server_base.cc:1034] running on GCE node
I20260810 14:32:48.852931 23311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:48.853828 23311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:48.855019 23311 hybrid_clock.cc:648] HybridClock initialized: now 1786372368854961 us; error 46 us; skew 500 ppm
I20260810 14:32:48.855307 23311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:32:48.857321 23311 webserver.cc:460] Webserver started at http://127.6.46.195:40735/ using document root <none> and password file <none>
I20260810 14:32:48.857959 23311 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:48.858088 23311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:48.858400 23311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:48.861056 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "948fbcb30a2445798b818b830bb821c5"
format_stamp: "Formatted at 2026-08-10 14:32:48 on dist-test-slave-q9h9"
server_key: "1a3c205f1f9124a89148edc3afc2caec"
server_key_iv: "48b67762e4c75cc47e6bf4f79e8ddd65"
server_key_version: "encryptionkey@0"
I20260810 14:32:48.861809 23311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "948fbcb30a2445798b818b830bb821c5"
format_stamp: "Formatted at 2026-08-10 14:32:48 on dist-test-slave-q9h9"
server_key: "1a3c205f1f9124a89148edc3afc2caec"
server_key_iv: "48b67762e4c75cc47e6bf4f79e8ddd65"
server_key_version: "encryptionkey@0"
I20260810 14:32:48.866696 23311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 14:32:48.869913 23334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:48.871150 23311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:32:48.871292 23311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "948fbcb30a2445798b818b830bb821c5"
format_stamp: "Formatted at 2026-08-10 14:32:48 on dist-test-slave-q9h9"
server_key: "1a3c205f1f9124a89148edc3afc2caec"
server_key_iv: "48b67762e4c75cc47e6bf4f79e8ddd65"
server_key_version: "encryptionkey@0"
I20260810 14:32:48.871488 23311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:48.876708 23311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:48.877434 23311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:48.877683 23311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:48.878615 23311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:48.879928 23311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:48.880023 23311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:48.880125 23311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:48.880196 23311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:48.909986 23311 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:43791
I20260810 14:32:48.910063 23525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:43791 every 8 connection(s)
I20260810 14:32:48.911329 23311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:32:48.918869 23086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:48.913789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:46377 (local address 127.6.46.254:38117)
0810 14:32:48.914034 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:48.914042 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:48.915354 (+ 1312us) server_negotiation.cc:407] Connection header received
0810 14:32:48.916946 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:48.916950 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:48.917016 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:48.917105 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:48.918292 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:48.918296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:48.918299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:48.918315 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:48.918348 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:48.918359 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:48.918455 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:48.918577 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 14:32:48.918629 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 14:32:48.919727 23528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:48.914299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:38117 (local address 127.6.46.195:46377)
0810 14:32:48.915205 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:48.915249 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:48.916711 (+ 1462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:48.917295 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:48.917303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:48.917389 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:48.917963 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:32:48.917966 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:48.918001 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:32:48.918119 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:48.918428 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:48.918434 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:48.918439 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:32:48.918638 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 14:32:48.918855 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":280,"threads_started":1}
I20260810 14:32:48.920972 23526 heartbeater.cc:347] Connected to a master server at 127.6.46.254:38117
I20260810 14:32:48.921279 23526 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:48.921603 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 23311
I20260810 14:32:48.921805 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:32:48.922036 23526 heartbeater.cc:511] Master 127.6.46.254:38117 requested a full tablet report, sending...
I20260810 14:32:48.922425 6331 external_mini_cluster.cc:1428] Setting key 30160a7535bb0e82bb62c7e985e8e0c6
I20260810 14:32:48.923616 22740 ts_manager.cc:194] Registered new tserver with Master: 948fbcb30a2445798b818b830bb821c5 (127.6.46.195:43791)
I20260810 14:32:48.925220 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 14:32:48.931005 23086 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:32:48.929532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40590 (local address 127.6.46.254:38117)
0810 14:32:48.929673 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:48.929678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:48.930187 (+ 509us) server_negotiation.cc:407] Connection header received
0810 14:32:48.930339 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:48.930343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:48.930391 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:48.930468 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:48.930799 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40590: BlockingRecv error: recv got EOF from 127.0.0.1:40590 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:32:48.931617 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 22855
I20260810 14:32:48.934574 23536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:48.932456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.6.46.193:38173)
0810 14:32:48.932906 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:48.932912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:48.932935 (+ 23us) server_negotiation.cc:407] Connection header received
0810 14:32:48.932988 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:48.932993 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:48.933149 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:48.933269 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:48.933988 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:48.933994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:48.934004 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:48.934048 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:48.934098 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:48.934115 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:48.934222 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:48.934293 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 14:32:48.934370 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":211,"threads_started":1}
I20260810 14:32:48.935657 23073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:49.014766 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 22855
I20260810 14:32:49.032426 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 23089
I20260810 14:32:49.036154 23538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:49.033372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53140 (local address 127.6.46.194:45091)
0810 14:32:49.033861 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:49.033868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:49.033895 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:32:49.033958 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:49.033964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:49.034238 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:49.034367 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:49.035357 (+ 990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:49.035363 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:49.035375 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:49.035387 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:49.035433 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:49.035451 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:49.035582 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:49.035699 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 14:32:49.035829 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":219,"threads_started":1}
I20260810 14:32:49.037621 23296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:49.116086 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 23089
I20260810 14:32:49.138918 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 23311
I20260810 14:32:49.142771 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:49.139836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.6.46.195:43791)
0810 14:32:49.140497 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:49.140505 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:49.140535 (+ 30us) server_negotiation.cc:407] Connection header received
0810 14:32:49.140598 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:49.140605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:49.140819 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:49.140938 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:49.141944 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:49.141951 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:49.141962 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:32:49.141980 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:49.142044 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:32:49.142066 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:49.142217 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:49.142346 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 14:32:49.142442 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":331,"threads_started":1}
I20260810 14:32:49.144095 23519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:49.224249 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 23311
I20260810 14:32:49.239984 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 22696
I20260810 14:32:49.241120 22805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:49.341194 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 22696
2026-08-10T14:32:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2271 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:32:49 dist-test-slave-q9h9 krb5kdc[23549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:32:49 dist-test-slave-q9h9 krb5kdc[23549](info): set up 2 sockets
Aug 10 14:32:49 dist-test-slave-q9h9 krb5kdc[23549](info): commencing operation
krb5kdc: starting...
W20260810 14:32:50.584249 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.193s user 0.006s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:32:50 dist-test-slave-q9h9 krb5kdc[23549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:32:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:32:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:32:50.820617 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:38359
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:38359
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:51.136533 23575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:51.136826 23575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:51.136997 23575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:51.144068 23575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:32:51.144160 23575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:51.144212 23575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:51.144268 23575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:32:51.144308 23575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:32:51.153723 23575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:38359
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:38359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:51.154963 23575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:51.156195 23575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:51.167255 23583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:51.167330 23586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:51.167294 23589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:51.167919 23575 server_base.cc:1034] running on GCE node
I20260810 14:32:51.168628 23575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:51.169940 23575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:51.171134 23575 hybrid_clock.cc:648] HybridClock initialized: now 1786372371171089 us; error 36 us; skew 500 ppm
I20260810 14:32:51.171446 23575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:51 dist-test-slave-q9h9 krb5kdc[23549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372371, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:51.179277 23575 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:32:51.180763 23575 webserver.cc:460] Webserver started at http://127.6.46.254:38639/ using document root <none> and password file <none>
I20260810 14:32:51.181432 23575 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:51.181555 23575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:51.181843 23575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:51.184780 23575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "c60f1a25f00542d99bb785cfba374602"
format_stamp: "Formatted at 2026-08-10 14:32:51 on dist-test-slave-q9h9"
server_key: "d004afa751e464bf972b1eda0b270069"
server_key_iv: "1472fd343dc9a21f87d5e77c8f039822"
server_key_version: "encryptionkey@0"
I20260810 14:32:51.185518 23575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "c60f1a25f00542d99bb785cfba374602"
format_stamp: "Formatted at 2026-08-10 14:32:51 on dist-test-slave-q9h9"
server_key: "d004afa751e464bf972b1eda0b270069"
server_key_iv: "1472fd343dc9a21f87d5e77c8f039822"
server_key_version: "encryptionkey@0"
I20260810 14:32:51.190917 23575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 14:32:51.194312 23602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:51.195611 23575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:32:51.195775 23575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "c60f1a25f00542d99bb785cfba374602"
format_stamp: "Formatted at 2026-08-10 14:32:51 on dist-test-slave-q9h9"
server_key: "d004afa751e464bf972b1eda0b270069"
server_key_iv: "1472fd343dc9a21f87d5e77c8f039822"
server_key_version: "encryptionkey@0"
I20260810 14:32:51.195959 23575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:51.215723 23575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:51.219521 23575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:51.219776 23575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:51.237226 23575 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:38359
I20260810 14:32:51.237246 23690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:38359 every 8 connection(s)
I20260810 14:32:51.238587 23575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:32:51.240831 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 23575
I20260810 14:32:51.241044 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:32:51.241887 6331 external_mini_cluster.cc:1428] Setting key fa2e858d7bce4e95bd0134f0210d2a43
I20260810 14:32:51.242785 23692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:51.254555 23692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602: Bootstrap starting.
Aug 10 14:32:51 dist-test-slave-q9h9 krb5kdc[23549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:32:51.257633 23692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:51.258705 23692 log.cc:824] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:51.261268 23692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602: No bootstrap required, opened a new log
I20260810 14:32:51.262604 23696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:51.243956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40782 (local address 127.6.46.254:38359)
0810 14:32:51.244735 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:51.244746 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:51.244786 (+ 40us) server_negotiation.cc:407] Connection header received
0810 14:32:51.246322 (+ 1536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:51.246343 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:51.246745 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:51.247114 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:51.248555 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:51.249864 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:51.251098 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:51.251775 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:51.251958 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:51.255916 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:51.255923 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:51.255936 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:51.255977 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:51.258884 (+ 2907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:51.259497 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:51.259503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:51.259509 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:51.259677 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:51.260051 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:51.260055 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:51.260057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:51.260547 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:51.260834 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:51.261256 (+ 422us) server_negotiation.cc:299] Negotiation successful
0810 14:32:51.261634 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":334,"threads_started":1}
I20260810 14:32:51.268896 23692 raft_consensus.cc:348] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60f1a25f00542d99bb785cfba374602" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38359 } }
I20260810 14:32:51.269214 23692 raft_consensus.cc:374] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:51.269304 23692 raft_consensus.cc:729] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60f1a25f00542d99bb785cfba374602, State: Initialized, Role: FOLLOWER
I20260810 14:32:51.269794 23692 consensus_queue.cc:260] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60f1a25f00542d99bb785cfba374602" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38359 } }
I20260810 14:32:51.270004 23692 raft_consensus.cc:388] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:32:51.270151 23692 raft_consensus.cc:482] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:32:51.270309 23692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:51.272291 23692 raft_consensus.cc:504] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60f1a25f00542d99bb785cfba374602" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38359 } }
I20260810 14:32:51.272744 23692 leader_election.cc:302] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c60f1a25f00542d99bb785cfba374602; no voters:
I20260810 14:32:51.273264 23692 leader_election.cc:288] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:32:51.273375 23698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:51.273998 23698 raft_consensus.cc:686] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [term 1 LEADER]: Becoming Leader. State: Replica: c60f1a25f00542d99bb785cfba374602, State: Running, Role: LEADER
I20260810 14:32:51.274509 23698 consensus_queue.cc:237] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60f1a25f00542d99bb785cfba374602" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38359 } }
I20260810 14:32:51.275137 23692 sys_catalog.cc:564] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:32:51.277242 23700 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c60f1a25f00542d99bb785cfba374602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60f1a25f00542d99bb785cfba374602" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38359 } } }
I20260810 14:32:51.277622 23700 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:51.278374 23702 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c60f1a25f00542d99bb785cfba374602. Latest consensus state: current_term: 1 leader_uuid: "c60f1a25f00542d99bb785cfba374602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60f1a25f00542d99bb785cfba374602" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 38359 } } }
I20260810 14:32:51.278702 23702 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602 [sys.catalog]: This master's current role is: LEADER
I20260810 14:32:51.279071 23721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:32:51.284201 23721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:32:51.291774 23721 catalog_manager.cc:1347] Generated new cluster ID: 4c5d1d958df14abaa0b650ec513ae0fa
I20260810 14:32:51.291874 23721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:32:51.310087 23721 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:32:51.311303 23721 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:32:51.321767 23721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602: Generated new TSK 0
I20260810 14:32:51.322599 23721 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:32:51.391047 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38359
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:51.738801 23745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:51.739063 23745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:51.739209 23745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:51.746202 23745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:51.746310 23745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:51.746465 23745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:32:51.755724 23745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:51.756889 23745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:51.758172 23745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:51.770846 23759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:51.770886 23755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:51.770910 23754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:51.771759 23745 server_base.cc:1034] running on GCE node
I20260810 14:32:51.772289 23745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:51.773125 23745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:51.774313 23745 hybrid_clock.cc:648] HybridClock initialized: now 1786372371774270 us; error 32 us; skew 500 ppm
I20260810 14:32:51.774607 23745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:51 dist-test-slave-q9h9 krb5kdc[23549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372371, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:51.777724 23745 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:32:51.779142 23745 webserver.cc:460] Webserver started at http://127.6.46.193:41005/ using document root <none> and password file <none>
I20260810 14:32:51.779830 23745 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:51.779937 23745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:51.780241 23745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:51.783042 23745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "5883cb4b186747a78f770bb68172afbf"
format_stamp: "Formatted at 2026-08-10 14:32:51 on dist-test-slave-q9h9"
server_key: "1524b4f4996b9ab00e732206edd30780"
server_key_iv: "5b72bf55ce020f3dd2086624f60adedc"
server_key_version: "encryptionkey@0"
I20260810 14:32:51.783775 23745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "5883cb4b186747a78f770bb68172afbf"
format_stamp: "Formatted at 2026-08-10 14:32:51 on dist-test-slave-q9h9"
server_key: "1524b4f4996b9ab00e732206edd30780"
server_key_iv: "5b72bf55ce020f3dd2086624f60adedc"
server_key_version: "encryptionkey@0"
I20260810 14:32:51.788581 23745 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:32:51.791863 23770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:51.793154 23745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:32:51.793309 23745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "5883cb4b186747a78f770bb68172afbf"
format_stamp: "Formatted at 2026-08-10 14:32:51 on dist-test-slave-q9h9"
server_key: "1524b4f4996b9ab00e732206edd30780"
server_key_iv: "5b72bf55ce020f3dd2086624f60adedc"
server_key_version: "encryptionkey@0"
I20260810 14:32:51.793506 23745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:51.809508 23745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:51.813169 23745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:51.813431 23745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:51.814361 23745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:51.815676 23745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:51.815752 23745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:51.815878 23745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:51.815928 23745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:51.845925 23745 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:43793
I20260810 14:32:51.845969 23976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:43793 every 8 connection(s)
I20260810 14:32:51.847162 23745 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:32:51.849987 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 23745
I20260810 14:32:51.850231 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:32:51.851060 6331 external_mini_cluster.cc:1428] Setting key 3f0e9edeb341b09a2459082cc7f92daa
Aug 10 14:32:51 dist-test-slave-q9h9 krb5kdc[23549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372371, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:32:51.865742 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:51.849506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:39419 (local address 127.6.46.254:38359)
0810 14:32:51.849955 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:51.849959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:51.850920 (+ 961us) server_negotiation.cc:407] Connection header received
0810 14:32:51.852905 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:51.852911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:51.852993 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:51.853100 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:51.854593 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:51.855315 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:51.856821 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:51.857201 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:51.857313 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:51.860517 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:51.860529 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:51.860533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:51.860565 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:51.862894 (+ 2329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:51.863571 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:51.863575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:51.863577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:51.863696 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:51.864105 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:51.864109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:51.864111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:51.864363 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:51.864499 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:51.865155 (+ 656us) server_negotiation.cc:299] Negotiation successful
0810 14:32:51.865377 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":233,"threads_started":1}
I20260810 14:32:51.866616 23983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:51.849973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:38359 (local address 127.6.46.193:39419)
0810 14:32:51.850759 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:51.850804 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:51.852632 (+ 1828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:51.853331 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:51.853338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:51.853825 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:51.854430 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:51.854448 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:51.855475 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:51.855481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:51.856676 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:51.856688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:51.857417 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:51.857425 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:51.857649 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:51.858264 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:51.858285 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:51.860308 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:51.863032 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:51.863040 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:51.863055 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:51.863438 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:51.863798 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:51.863803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:51.863807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:51.863984 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:51.864489 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:51.864496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:51.864796 (+ 300us) client_negotiation.cc:769] Sending connection context
0810 14:32:51.865017 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 14:32:51.865496 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":311,"threads_started":1}
I20260810 14:32:51.868120 23978 heartbeater.cc:347] Connected to a master server at 127.6.46.254:38359
I20260810 14:32:51.868418 23978 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:51.869297 23978 heartbeater.cc:511] Master 127.6.46.254:38359 requested a full tablet report, sending...
I20260810 14:32:51.871539 23634 ts_manager.cc:194] Registered new tserver with Master: 5883cb4b186747a78f770bb68172afbf (127.6.46.193:43793)
I20260810 14:32:51.873378 23634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:39419
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:32:51.908373 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38359
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:52.224020 23989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:52.224272 23989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:52.224383 23989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:52.231295 23989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:52.231388 23989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:52.231530 23989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:32:52.240798 23989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:52.241865 23989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:52.243152 23989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:52.255168 24011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:52.255205 24002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:52.255237 24004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:52.256278 23989 server_base.cc:1034] running on GCE node
I20260810 14:32:52.256834 23989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:52.257654 23989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:52.258841 23989 hybrid_clock.cc:648] HybridClock initialized: now 1786372372258800 us; error 30 us; skew 500 ppm
I20260810 14:32:52.259121 23989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:52 dist-test-slave-q9h9 krb5kdc[23549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372372, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:52.262118 23989 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:32:52.263381 23989 webserver.cc:460] Webserver started at http://127.6.46.194:38303/ using document root <none> and password file <none>
I20260810 14:32:52.264005 23989 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:52.264119 23989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:52.264402 23989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:52.267146 23989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "2f345819b39f4116899092fa35461f66"
format_stamp: "Formatted at 2026-08-10 14:32:52 on dist-test-slave-q9h9"
server_key: "b16c7583739440f875df256f5aa11955"
server_key_iv: "e332d9905c922d6f587c654801c8a2ba"
server_key_version: "encryptionkey@0"
I20260810 14:32:52.267874 23989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "2f345819b39f4116899092fa35461f66"
format_stamp: "Formatted at 2026-08-10 14:32:52 on dist-test-slave-q9h9"
server_key: "b16c7583739440f875df256f5aa11955"
server_key_iv: "e332d9905c922d6f587c654801c8a2ba"
server_key_version: "encryptionkey@0"
I20260810 14:32:52.272514 23989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:32:52.275797 24018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:52.276998 23989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:32:52.277143 23989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "2f345819b39f4116899092fa35461f66"
format_stamp: "Formatted at 2026-08-10 14:32:52 on dist-test-slave-q9h9"
server_key: "b16c7583739440f875df256f5aa11955"
server_key_iv: "e332d9905c922d6f587c654801c8a2ba"
server_key_version: "encryptionkey@0"
I20260810 14:32:52.277319 23989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:52.302989 23989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:52.306878 23989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:52.307129 23989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:52.307999 23989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:52.309252 23989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:52.309329 23989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:52.309442 23989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:52.309495 23989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:52.338697 23989 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:36917
I20260810 14:32:52.338758 24207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:36917 every 8 connection(s)
I20260810 14:32:52.339920 23989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:32:52.340758 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 23989
I20260810 14:32:52.340934 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:32:52.341436 6331 external_mini_cluster.cc:1428] Setting key 9b465fa959be6ad25ff50f45708b337f
Aug 10 14:32:52 dist-test-slave-q9h9 krb5kdc[23549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372372, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:32:52.359202 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:52.342362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:33393 (local address 127.6.46.254:38359)
0810 14:32:52.342543 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:52.342549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:52.344019 (+ 1470us) server_negotiation.cc:407] Connection header received
0810 14:32:52.345804 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:52.345809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:52.345886 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:52.345981 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:52.347416 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.348298 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.349813 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.350397 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.350515 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:52.353651 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:52.353656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:52.353661 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:52.353695 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:52.356322 (+ 2627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:52.357154 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:52.357157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:52.357159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:52.357276 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:52.357693 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:52.357696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:52.357698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:52.357930 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:52.358067 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:52.358797 (+ 730us) server_negotiation.cc:299] Negotiation successful
0810 14:32:52.358968 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:32:52.360487 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:52.342922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:38359 (local address 127.6.46.194:33393)
0810 14:32:52.343848 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:52.343902 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:52.345546 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:52.346222 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:52.346230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:52.346709 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:52.347253 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:52.347271 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.348420 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.348424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:52.349671 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:52.349684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.350576 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.350582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:52.350766 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:52.351371 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:52.351393 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:52.353434 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:52.356457 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:52.356466 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:52.356480 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:52.357059 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:52.357369 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:52.357374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:52.357378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:52.357599 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:52.358078 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:52.358085 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:52.358451 (+ 366us) client_negotiation.cc:769] Sending connection context
0810 14:32:52.358726 (+ 275us) client_negotiation.cc:240] Negotiation successful
0810 14:32:52.359206 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":478,"threads_started":1}
I20260810 14:32:52.361816 24208 heartbeater.cc:347] Connected to a master server at 127.6.46.254:38359
I20260810 14:32:52.362138 24208 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:52.362813 24208 heartbeater.cc:511] Master 127.6.46.254:38359 requested a full tablet report, sending...
I20260810 14:32:52.364253 23634 ts_manager.cc:194] Registered new tserver with Master: 2f345819b39f4116899092fa35461f66 (127.6.46.194:36917)
I20260810 14:32:52.365025 23634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:33393
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:32:52.398532 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38359
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:44435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:32:52.717736 24220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:32:52.717998 24220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:32:52.718159 24220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:32:52.725117 24220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:32:52.725224 24220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:32:52.725373 24220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:32:52.734540 24220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:44435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:38359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:32:52.735679 24220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:32:52.736901 24220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:32:52.749171 24232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:52.749202 24230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:32:52.749131 24234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:32:52.750185 24220 server_base.cc:1034] running on GCE node
I20260810 14:32:52.750777 24220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:32:52.751616 24220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:32:52.752818 24220 hybrid_clock.cc:648] HybridClock initialized: now 1786372372752768 us; error 43 us; skew 500 ppm
I20260810 14:32:52.753108 24220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:32:52 dist-test-slave-q9h9 krb5kdc[23549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372372, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:32:52.756110 24220 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:32:52.757417 24220 webserver.cc:460] Webserver started at http://127.6.46.195:34447/ using document root <none> and password file <none>
I20260810 14:32:52.758080 24220 fs_manager.cc:359] Metadata directory not provided
I20260810 14:32:52.758200 24220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:32:52.758473 24220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:32:52.761184 24220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "bc445109a27c46e1bca9b854cd4380a2"
format_stamp: "Formatted at 2026-08-10 14:32:52 on dist-test-slave-q9h9"
server_key: "4c06469d4c418785b7bfdd61357850fb"
server_key_iv: "82cba46230ce67dd9047816e51fb6baa"
server_key_version: "encryptionkey@0"
I20260810 14:32:52.761919 24220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "bc445109a27c46e1bca9b854cd4380a2"
format_stamp: "Formatted at 2026-08-10 14:32:52 on dist-test-slave-q9h9"
server_key: "4c06469d4c418785b7bfdd61357850fb"
server_key_iv: "82cba46230ce67dd9047816e51fb6baa"
server_key_version: "encryptionkey@0"
I20260810 14:32:52.766901 24220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 14:32:52.770153 24243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:52.771350 24220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:32:52.771500 24220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "bc445109a27c46e1bca9b854cd4380a2"
format_stamp: "Formatted at 2026-08-10 14:32:52 on dist-test-slave-q9h9"
server_key: "4c06469d4c418785b7bfdd61357850fb"
server_key_iv: "82cba46230ce67dd9047816e51fb6baa"
server_key_version: "encryptionkey@0"
I20260810 14:32:52.771682 24220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:32:52.789120 24220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:32:52.793037 24220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:32:52.793294 24220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:32:52.794227 24220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:32:52.795497 24220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:32:52.795575 24220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:52.795694 24220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:32:52.795754 24220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:32:52.825627 24220 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:43991
I20260810 14:32:52.825681 24423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:43991 every 8 connection(s)
I20260810 14:32:52.826864 24220 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:32:52.829619 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 24220
I20260810 14:32:52.829772 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:32:52.830350 6331 external_mini_cluster.cc:1428] Setting key 662c6cb7666badaf9d95f74b1f527ad1
Aug 10 14:32:52 dist-test-slave-q9h9 krb5kdc[23549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372372, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:32:52.845758 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:52.829251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:49373 (local address 127.6.46.254:38359)
0810 14:32:52.829489 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:52.829495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:52.830670 (+ 1175us) server_negotiation.cc:407] Connection header received
0810 14:32:52.832734 (+ 2064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:52.832738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:52.832805 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:52.832883 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:52.834446 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.835209 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.836617 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.837024 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.837116 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:52.840205 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:52.840210 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:52.840213 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:52.840247 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:52.842759 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:52.843395 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:52.843400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:52.843404 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:52.843555 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:52.844023 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:52.844028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:52.844036 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:52.844316 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:52.844465 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:52.845251 (+ 786us) server_negotiation.cc:299] Negotiation successful
0810 14:32:52.845426 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 14:32:52.846827 24428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:52.829656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:38359 (local address 127.6.46.195:49373)
0810 14:32:52.830493 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:52.830541 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:52.832453 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:52.833136 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:52.833148 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:52.833727 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:32:52.834299 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:52.834317 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.835321 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.835325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:52.836490 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:52.836501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.837189 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.837193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:52.837375 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:52.837907 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:32:52.837931 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:32:52.840011 (+ 2080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:32:52.842864 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:52.842870 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:52.842880 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:52.843256 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:52.843659 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:32:52.843663 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:32:52.843666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:32:52.843901 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:32:52.844455 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:32:52.844462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:32:52.844825 (+ 363us) client_negotiation.cc:769] Sending connection context
0810 14:32:52.845115 (+ 290us) client_negotiation.cc:240] Negotiation successful
0810 14:32:52.845579 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":323,"threads_started":1}
I20260810 14:32:52.848052 24425 heartbeater.cc:347] Connected to a master server at 127.6.46.254:38359
I20260810 14:32:52.848358 24425 heartbeater.cc:464] Registering TS with master...
I20260810 14:32:52.849206 24425 heartbeater.cc:511] Master 127.6.46.254:38359 requested a full tablet report, sending...
I20260810 14:32:52.850703 23634 ts_manager.cc:194] Registered new tserver with Master: bc445109a27c46e1bca9b854cd4380a2 (127.6.46.195:43991)
I20260810 14:32:52.851538 23634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:49373
I20260810 14:32:52.857707 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:32:52.873849 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:52.862061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40792 (local address 127.6.46.254:38359)
0810 14:32:52.862206 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:52.862211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:52.862647 (+ 436us) server_negotiation.cc:407] Connection header received
0810 14:32:52.862807 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:52.862811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:52.862880 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:52.862962 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:52.864369 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.865066 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.866305 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.866710 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.866820 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:52.868957 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:52.868961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:52.868964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:52.868996 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:52.871394 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:52.871986 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:52.871991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:52.871993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:52.872125 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:52.872490 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:52.872494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:52.872496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:52.872799 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:52.873033 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:52.873372 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 14:32:52.873585 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:32:52.876742 23978 heartbeater.cc:503] Master 127.6.46.254:38359 was elected leader, sending a full tablet report...
I20260810 14:32:52.877990 23634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:32:52.884850 23634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:32:52.908197 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:52.896057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:36917 (local address 127.0.0.1:52648)
0810 14:32:52.897139 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:52.897183 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:52.897412 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:52.898855 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:52.898863 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:52.898899 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:52.899344 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:52.899356 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.901865 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.901872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:52.904381 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:52.904399 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.907178 (+ 2779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.907184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:52.907423 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:52.907481 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 14:32:52.907602 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 14:32:52.907795 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":10432,"threads_started":1}
I20260810 14:32:52.909690 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:52.896389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.6.46.194:36917)
0810 14:32:52.898135 (+ 1746us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:52.898145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:52.898173 (+ 28us) server_negotiation.cc:407] Connection header received
0810 14:32:52.898269 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:52.898277 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:52.898554 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:52.899610 (+ 1056us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:52.899871 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.901689 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.904626 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.906243 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.908837 (+ 2594us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:52.909068 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:52.909138 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 14:32:52.909321 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1561,"thread_start_us":1420,"threads_started":1}
I20260810 14:32:52.913839 24105 tablet_service.cc:1467] Processing CreateTablet for tablet c372c9dfef29458d9cfbab95524b0a93 (DEFAULT_TABLE table=test-table [id=62661cee48014a8b8f2b230d8dfe7ea9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:52.915388 24105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c372c9dfef29458d9cfbab95524b0a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:52.916369 24446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:52.898094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:43793 (local address 127.0.0.1:42326)
0810 14:32:52.908554 (+ 10460us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:52.908573 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:52.908671 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:52.910349 (+ 1678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:52.910354 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:52.910378 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:52.910538 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:52.910550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.912538 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.912543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:52.914363 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:52.914375 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.915640 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.915646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:52.915829 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:52.915852 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:32:52.915928 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 14:32:52.916082 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10354,"thread_start_us":12025,"threads_started":1}
I20260810 14:32:52.916417 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:52.898344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42326 (local address 127.6.46.193:43793)
0810 14:32:52.909153 (+ 10809us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:52.909163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:52.909194 (+ 31us) server_negotiation.cc:407] Connection header received
0810 14:32:52.909299 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:52.909308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:52.909558 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:52.910701 (+ 1143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:52.910901 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.912388 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.914503 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.915430 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.915550 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:52.915777 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:52.915985 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 14:32:52.916132 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10626,"thread_start_us":11371,"threads_started":1}
I20260810 14:32:52.918819 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:52.896447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:43991 (local address 127.0.0.1:36416)
0810 14:32:52.906808 (+ 10361us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:52.906829 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:52.906947 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:52.911814 (+ 4867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:52.911818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:52.911841 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:52.912168 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:52.912178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.914072 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.914078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:52.916026 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:52.916041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.918094 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.918101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:52.918281 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:52.918312 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 14:32:52.918405 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 14:32:52.918524 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10199,"thread_start_us":344,"threads_started":1}
I20260810 14:32:52.918929 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:52.896795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36416 (local address 127.6.46.195:43991)
0810 14:32:52.911388 (+ 14593us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:52.911394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:52.911418 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:32:52.911491 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:52.911496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:52.911647 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:52.911806 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:52.912323 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.913906 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.916587 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:52.917893 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:52.918038 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:52.918245 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:52.918476 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 14:32:52.918636 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14452,"thread_start_us":14320,"threads_started":1}
I20260810 14:32:52.920071 23860 tablet_service.cc:1467] Processing CreateTablet for tablet c372c9dfef29458d9cfbab95524b0a93 (DEFAULT_TABLE table=test-table [id=62661cee48014a8b8f2b230d8dfe7ea9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:52.921452 23860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c372c9dfef29458d9cfbab95524b0a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:52.922585 24318 tablet_service.cc:1467] Processing CreateTablet for tablet c372c9dfef29458d9cfbab95524b0a93 (DEFAULT_TABLE table=test-table [id=62661cee48014a8b8f2b230d8dfe7ea9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:32:52.923908 24318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c372c9dfef29458d9cfbab95524b0a93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:32:52.928179 24451 tablet_bootstrap.cc:492] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66: Bootstrap starting.
I20260810 14:32:52.931186 24451 tablet_bootstrap.cc:654] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:52.932302 24451 log.cc:824] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:52.935456 24455 tablet_bootstrap.cc:492] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf: Bootstrap starting.
I20260810 14:32:52.935683 24451 tablet_bootstrap.cc:492] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66: No bootstrap required, opened a new log
I20260810 14:32:52.936003 24451 ts_tablet_manager.cc:1397] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 14:32:52.937348 24458 tablet_bootstrap.cc:492] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2: Bootstrap starting.
I20260810 14:32:52.938917 24455 tablet_bootstrap.cc:654] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:52.940135 24455 log.cc:824] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:52.940232 24458 tablet_bootstrap.cc:654] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2: Neither blocks nor log segments found. Creating new log.
I20260810 14:32:52.941491 24458 log.cc:824] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2: Log is configured to *not* fsync() on all Append() calls
I20260810 14:32:52.943226 24455 tablet_bootstrap.cc:492] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf: No bootstrap required, opened a new log
I20260810 14:32:52.943599 24455 ts_tablet_manager.cc:1397] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 14:32:52.944157 24458 tablet_bootstrap.cc:492] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2: No bootstrap required, opened a new log
I20260810 14:32:52.944393 24458 ts_tablet_manager.cc:1397] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260810 14:32:52.946121 24451 raft_consensus.cc:348] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f345819b39f4116899092fa35461f66" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36917 } } peers { permanent_uuid: "bc445109a27c46e1bca9b854cd4380a2" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43991 } } peers { permanent_uuid: "5883cb4b186747a78f770bb68172afbf" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 43793 } }
I20260810 14:32:52.946545 24451 raft_consensus.cc:374] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:52.946640 24451 raft_consensus.cc:729] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f345819b39f4116899092fa35461f66, State: Initialized, Role: FOLLOWER
I20260810 14:32:52.947279 24451 consensus_queue.cc:260] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f345819b39f4116899092fa35461f66" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36917 } } peers { permanent_uuid: "bc445109a27c46e1bca9b854cd4380a2" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43991 } } peers { permanent_uuid: "5883cb4b186747a78f770bb68172afbf" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 43793 } }
I20260810 14:32:52.948935 24451 ts_tablet_manager.cc:1428] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260810 14:32:52.949398 24208 heartbeater.cc:503] Master 127.6.46.254:38359 was elected leader, sending a full tablet report...
I20260810 14:32:52.952791 24455 raft_consensus.cc:348] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f345819b39f4116899092fa35461f66" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36917 } } peers { permanent_uuid: "bc445109a27c46e1bca9b854cd4380a2" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43991 } } peers { permanent_uuid: "5883cb4b186747a78f770bb68172afbf" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 43793 } }
I20260810 14:32:52.953205 24455 raft_consensus.cc:374] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:52.953300 24455 raft_consensus.cc:729] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5883cb4b186747a78f770bb68172afbf, State: Initialized, Role: FOLLOWER
I20260810 14:32:52.953043 24458 raft_consensus.cc:348] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f345819b39f4116899092fa35461f66" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36917 } } peers { permanent_uuid: "bc445109a27c46e1bca9b854cd4380a2" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43991 } } peers { permanent_uuid: "5883cb4b186747a78f770bb68172afbf" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 43793 } }
I20260810 14:32:52.953975 24458 raft_consensus.cc:374] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:32:52.953845 24455 consensus_queue.cc:260] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f345819b39f4116899092fa35461f66" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36917 } } peers { permanent_uuid: "bc445109a27c46e1bca9b854cd4380a2" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43991 } } peers { permanent_uuid: "5883cb4b186747a78f770bb68172afbf" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 43793 } }
I20260810 14:32:52.954102 24458 raft_consensus.cc:729] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc445109a27c46e1bca9b854cd4380a2, State: Initialized, Role: FOLLOWER
I20260810 14:32:52.955396 24455 ts_tablet_manager.cc:1428] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260810 14:32:52.955673 24458 consensus_queue.cc:260] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f345819b39f4116899092fa35461f66" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36917 } } peers { permanent_uuid: "bc445109a27c46e1bca9b854cd4380a2" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43991 } } peers { permanent_uuid: "5883cb4b186747a78f770bb68172afbf" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 43793 } }
I20260810 14:32:52.956736 24425 heartbeater.cc:503] Master 127.6.46.254:38359 was elected leader, sending a full tablet report...
I20260810 14:32:52.957212 24458 ts_tablet_manager.cc:1428] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260810 14:32:53.076850 24466 raft_consensus.cc:482] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:32:53.077152 24466 raft_consensus.cc:504] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f345819b39f4116899092fa35461f66" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36917 } } peers { permanent_uuid: "bc445109a27c46e1bca9b854cd4380a2" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43991 } } peers { permanent_uuid: "5883cb4b186747a78f770bb68172afbf" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 43793 } }
W20260810 14:32:53.078343 24426 tablet.cc:2369] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:32:53.079214 24466 leader_election.cc:288] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc445109a27c46e1bca9b854cd4380a2 (127.6.46.195:43991), 5883cb4b186747a78f770bb68172afbf (127.6.46.193:43793)
I20260810 14:32:53.085590 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:53.079546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:50727 (local address 127.6.46.195:43991)
0810 14:32:53.079711 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:53.079716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:53.080178 (+ 462us) server_negotiation.cc:407] Connection header received
0810 14:32:53.080485 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:53.080491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:53.080582 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:53.080677 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:53.081460 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.082283 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.083994 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.084741 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.084891 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:53.084965 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:53.085230 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 14:32:53.085352 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:32:53.085744 24469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:53.079454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:43991 (local address 127.6.46.194:50727)
0810 14:32:53.080068 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:53.080087 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:53.080300 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:53.080863 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:53.080868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:53.080926 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:53.081314 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:53.081347 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.082408 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.082413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:53.083850 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:53.083863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.084941 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.084948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:53.085090 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:53.085114 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:32:53.085193 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:32:53.085314 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":1,"thread_start_us":312,"threads_started":1}
I20260810 14:32:53.086190 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:53.080538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:37303 (local address 127.6.46.193:43793)
0810 14:32:53.080784 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:53.080790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:53.081182 (+ 392us) server_negotiation.cc:407] Connection header received
0810 14:32:53.081342 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:53.081347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:53.081420 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:53.081496 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:32:53.082141 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.082815 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.084471 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.085242 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.085390 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:53.085459 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:53.085812 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 14:32:53.085941 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 14:32:53.086675 24470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:53.080277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:43793 (local address 127.6.46.194:37303)
0810 14:32:53.081073 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0810 14:32:53.081089 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:32:53.081216 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:32:53.081543 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:32:53.081548 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:32:53.081576 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:32:53.081941 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:53.081951 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.082961 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.082967 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:53.084310 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:32:53.084323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.085463 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.085470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:32:53.085623 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:53.085648 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:32:53.086358 (+ 710us) client_negotiation.cc:240] Negotiation successful
0810 14:32:53.086455 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":483,"threads_started":1}
I20260810 14:32:53.086683 24350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c372c9dfef29458d9cfbab95524b0a93" candidate_uuid: "2f345819b39f4116899092fa35461f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc445109a27c46e1bca9b854cd4380a2" is_pre_election: true
I20260810 14:32:53.087218 24350 raft_consensus.cc:2454] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f345819b39f4116899092fa35461f66 in term 0.
I20260810 14:32:53.087682 23900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c372c9dfef29458d9cfbab95524b0a93" candidate_uuid: "2f345819b39f4116899092fa35461f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5883cb4b186747a78f770bb68172afbf" is_pre_election: true
I20260810 14:32:53.087899 24021 leader_election.cc:302] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f345819b39f4116899092fa35461f66, bc445109a27c46e1bca9b854cd4380a2; no voters:
I20260810 14:32:53.088150 23900 raft_consensus.cc:2454] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f345819b39f4116899092fa35461f66 in term 0.
I20260810 14:32:53.088299 24466 raft_consensus.cc:2781] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:32:53.088441 24466 raft_consensus.cc:482] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:32:53.088551 24466 raft_consensus.cc:3037] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:53.090771 24466 raft_consensus.cc:504] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f345819b39f4116899092fa35461f66" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36917 } } peers { permanent_uuid: "bc445109a27c46e1bca9b854cd4380a2" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43991 } } peers { permanent_uuid: "5883cb4b186747a78f770bb68172afbf" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 43793 } }
I20260810 14:32:53.091384 24466 leader_election.cc:288] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [CANDIDATE]: Term 1 election: Requested vote from peers bc445109a27c46e1bca9b854cd4380a2 (127.6.46.195:43991), 5883cb4b186747a78f770bb68172afbf (127.6.46.193:43793)
I20260810 14:32:53.091780 24350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c372c9dfef29458d9cfbab95524b0a93" candidate_uuid: "2f345819b39f4116899092fa35461f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc445109a27c46e1bca9b854cd4380a2"
W20260810 14:32:53.091845 24209 tablet.cc:2369] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:32:53.091892 23900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c372c9dfef29458d9cfbab95524b0a93" candidate_uuid: "2f345819b39f4116899092fa35461f66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5883cb4b186747a78f770bb68172afbf"
I20260810 14:32:53.091993 24350 raft_consensus.cc:3037] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:53.092087 23900 raft_consensus.cc:3037] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:32:53.095127 24350 raft_consensus.cc:2454] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f345819b39f4116899092fa35461f66 in term 1.
I20260810 14:32:53.095127 23900 raft_consensus.cc:2454] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f345819b39f4116899092fa35461f66 in term 1.
I20260810 14:32:53.095479 24021 leader_election.cc:302] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f345819b39f4116899092fa35461f66, 5883cb4b186747a78f770bb68172afbf; no voters:
I20260810 14:32:53.095734 24466 raft_consensus.cc:2781] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:32:53.096153 24466 raft_consensus.cc:686] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [term 1 LEADER]: Becoming Leader. State: Replica: 2f345819b39f4116899092fa35461f66, State: Running, Role: LEADER
I20260810 14:32:53.096618 24466 consensus_queue.cc:237] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f345819b39f4116899092fa35461f66" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36917 } } peers { permanent_uuid: "bc445109a27c46e1bca9b854cd4380a2" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43991 } } peers { permanent_uuid: "5883cb4b186747a78f770bb68172afbf" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 43793 } }
W20260810 14:32:53.099381 23981 tablet.cc:2369] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:32:53.101078 23633 catalog_manager.cc:5495] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f345819b39f4116899092fa35461f66 (127.6.46.194). New cstate: current_term: 1 leader_uuid: "2f345819b39f4116899092fa35461f66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f345819b39f4116899092fa35461f66" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36917 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc445109a27c46e1bca9b854cd4380a2" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5883cb4b186747a78f770bb68172afbf" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 43793 } health_report { overall_health: UNKNOWN } } }
I20260810 14:32:53.155886 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:53.150912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52654 (local address 127.6.46.194:36917)
0810 14:32:53.151134 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:53.151138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:53.151153 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:32:53.151236 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:53.151240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:53.151313 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:53.151440 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:32:53.151817 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.152525 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.153830 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.154257 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.154370 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:53.154847 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:32:53.155310 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:32:53.155440 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:53.155559 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 14:32:53.155638 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:32:53.163910 24350 raft_consensus.cc:1261] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2 [term 1 FOLLOWER]: Refusing update from remote peer 2f345819b39f4116899092fa35461f66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:53.164083 23900 raft_consensus.cc:1261] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf [term 1 FOLLOWER]: Refusing update from remote peer 2f345819b39f4116899092fa35461f66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:32:53.164831 24472 consensus_queue.cc:1035] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc445109a27c46e1bca9b854cd4380a2" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:53.165205 24466 consensus_queue.cc:1035] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5883cb4b186747a78f770bb68172afbf" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 43793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:32:53.188865 23634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:53.189149 23634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:32:53.192761 23634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c60f1a25f00542d99bb785cfba374602: Sending DeleteTablet for 3 replicas of tablet c372c9dfef29458d9cfbab95524b0a93
I20260810 14:32:53.194036 23860 tablet_service.cc:1514] Processing DeleteTablet for tablet c372c9dfef29458d9cfbab95524b0a93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:53 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:42326
I20260810 14:32:53.194087 24105 tablet_service.cc:1514] Processing DeleteTablet for tablet c372c9dfef29458d9cfbab95524b0a93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:53 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:52648
I20260810 14:32:53.195019 24318 tablet_service.cc:1514] Processing DeleteTablet for tablet c372c9dfef29458d9cfbab95524b0a93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:32:53 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:36416
I20260810 14:32:53.195500 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 23745
I20260810 14:32:53.207144 24501 tablet_replica.cc:330] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66: stopping tablet replica
I20260810 14:32:53.207615 24501 raft_consensus.cc:2229] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:32:53.208432 24501 raft_consensus.cc:2258] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:53.210412 24499 tablet_replica.cc:330] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf: stopping tablet replica
I20260810 14:32:53.210721 24499 raft_consensus.cc:2229] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:53.210830 24501 ts_tablet_manager.cc:1905] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:53.211135 24499 raft_consensus.cc:2258] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:53.212993 24499 ts_tablet_manager.cc:1905] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:53.216207 24502 tablet_replica.cc:330] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2: stopping tablet replica
Aug 10 14:32:53 dist-test-slave-q9h9 krb5kdc[23549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:32:53.216607 24502 raft_consensus.cc:2229] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:32:53.217110 24502 raft_consensus.cc:2258] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:32:53.219028 24501 ts_tablet_manager.cc:1918] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:53.219218 24501 log.cc:1196] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/c372c9dfef29458d9cfbab95524b0a93
I20260810 14:32:53.219399 24502 ts_tablet_manager.cc:1905] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:32:53.219470 24499 ts_tablet_manager.cc:1918] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:53.219633 24499 log.cc:1196] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/c372c9dfef29458d9cfbab95524b0a93
I20260810 14:32:53.220085 24501 ts_tablet_manager.cc:1939] T c372c9dfef29458d9cfbab95524b0a93 P 2f345819b39f4116899092fa35461f66: Deleting consensus metadata
I20260810 14:32:53.220134 24499 ts_tablet_manager.cc:1939] T c372c9dfef29458d9cfbab95524b0a93 P 5883cb4b186747a78f770bb68172afbf: Deleting consensus metadata
I20260810 14:32:53.221822 23606 catalog_manager.cc:4841] TS 5883cb4b186747a78f770bb68172afbf (127.6.46.193:43793): tablet c372c9dfef29458d9cfbab95524b0a93 (table test-table [id=62661cee48014a8b8f2b230d8dfe7ea9]) successfully deleted
I20260810 14:32:53.221853 23605 catalog_manager.cc:4841] TS 2f345819b39f4116899092fa35461f66 (127.6.46.194:36917): tablet c372c9dfef29458d9cfbab95524b0a93 (table test-table [id=62661cee48014a8b8f2b230d8dfe7ea9]) successfully deleted
I20260810 14:32:53.225000 24449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:53.196826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.6.46.193:43793)
0810 14:32:53.197042 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:53.197049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:53.206325 (+ 9276us) server_negotiation.cc:407] Connection header received
0810 14:32:53.206520 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:53.206527 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:53.206623 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:53.206733 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:53.208712 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.209894 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.211721 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.212326 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.212450 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:53.217772 (+ 5322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:53.217786 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:53.217803 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:53.217849 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:53.221651 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:53.222581 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:53.222590 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:53.222594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:53.222774 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:53.223230 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:53.223236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:53.223239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:53.223713 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:53.224037 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:53.224339 (+ 302us) server_negotiation.cc:299] Negotiation successful
0810 14:32:53.224558 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 14:32:53.225661 23970 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:53.225929 24502 ts_tablet_manager.cc:1918] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:32:53.226147 24502 log.cc:1196] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/c372c9dfef29458d9cfbab95524b0a93
I20260810 14:32:53.226763 24502 ts_tablet_manager.cc:1939] T c372c9dfef29458d9cfbab95524b0a93 P bc445109a27c46e1bca9b854cd4380a2: Deleting consensus metadata
I20260810 14:32:53.229696 23606 catalog_manager.cc:4841] TS bc445109a27c46e1bca9b854cd4380a2 (127.6.46.195:43991): tablet c372c9dfef29458d9cfbab95524b0a93 (table test-table [id=62661cee48014a8b8f2b230d8dfe7ea9]) successfully deleted
I20260810 14:32:53.313655 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 23745
I20260810 14:32:53.346999 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 23989
Aug 10 14:32:53 dist-test-slave-q9h9 krb5kdc[23549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:32:53.365339 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:53.348123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52668 (local address 127.6.46.194:36917)
0810 14:32:53.348290 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:53.348296 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:53.348313 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:32:53.348371 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:53.348376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:53.348454 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:53.348578 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:53.350173 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.351431 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.353205 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.353872 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.353995 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:53.359145 (+ 5150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:53.359159 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:53.359175 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:53.359221 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:53.362781 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:53.363373 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:53.363379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:53.363381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:53.363513 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:53.363865 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:53.363868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:53.363870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:53.364225 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:53.364534 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:53.364818 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 14:32:53.365027 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:32:53.366148 24202 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:53.474538 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 23989
I20260810 14:32:53.499394 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 24220
Aug 10 14:32:53 dist-test-slave-q9h9 krb5kdc[23549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:32:53.513767 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:32:53.500125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.6.46.195:43991)
0810 14:32:53.500292 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:32:53.500298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:32:53.500355 (+ 57us) server_negotiation.cc:407] Connection header received
0810 14:32:53.500549 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:32:53.500560 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:32:53.500626 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:32:53.500709 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:32:53.501968 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.502714 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.503926 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:32:53.504299 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:32:53.504408 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:32:53.508532 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:32:53.508545 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:32:53.508567 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:32:53.508604 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:32:53.511220 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:53.511834 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:53.511841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:53.511843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:53.511982 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:32:53.512325 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:32:53.512328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:32:53.512330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:32:53.512681 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:32:53.512960 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 14:32:53.513249 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 14:32:53.513451 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:32:53.514392 24417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:53.600497 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 24220
I20260810 14:32:53.625267 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 23575
I20260810 14:32:53.626667 23684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:32:53.735368 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 23575
2026-08-10T14:32:53Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4392 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:32:53 dist-test-slave-q9h9 krb5kdc[24511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:32:53 dist-test-slave-q9h9 krb5kdc[24511](info): set up 2 sockets
Aug 10 14:32:53 dist-test-slave-q9h9 krb5kdc[24511](info): commencing operation
krb5kdc: starting...
W20260810 14:32:54.966460 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:32:55 dist-test-slave-q9h9 krb5kdc[24511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:32:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:32:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/rangeradmin_127.6.46.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/rangeradmin_127.6.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/rangerlookup_127.6.46.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/rangerlookup_127.6.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/HTTP_127.6.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/HTTP_127.6.46.212@KRBTEST.COM.keytab.
I20260810 14:32:55.310952 6331 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task50NkZQ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/postgres -l logfile start
2026-08-10 14:32:57.509 UTC [24566] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 14:32:57.509 UTC [24566] LOG: listening on IPv4 address "127.6.46.212", port 52809
2026-08-10 14:32:57.512 UTC [24566] LOG: listening on Unix socket "/tmp/.s.PGSQL.52809"
2026-08-10 14:32:57.533 UTC [24570] LOG: database system was shut down at 2026-08-10 14:32:56 UTC
2026-08-10 14:32:57.536 UTC [24566] LOG: database system is ready to accept connections
I20260810 14:32:58.632114 6331 mini_postgres.cc:96] Postgres bound to 52809
2026-08-10 14:32:58.657 UTC [24583] FATAL: database "slave" does not exist
127.6.46.212:52809 - accepting connections
I20260810 14:32:58.658311 6331 mini_ranger.cc:161] Starting Ranger...
I20260810 14:32:58.685877 6331 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 14:32:59.077836 6331 mini_ranger.cc:87] Created ranger Postgres database
I20260810 14:32:59.077977 6331 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-admin
2026-08-10 14:32:59,416 [I] DB FLAVOR :POSTGRES
2026-08-10 14:32:59,417 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:32:59,417 [I] Checking connection..
2026-08-10 14:32:59,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 14:32:59,677 [I] Checking connection passed.
2026-08-10 14:32:59,677 [I] --------- Verifying version history table ---------
2026-08-10 14:32:59,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:32:59,930 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:32:59,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:33:00,161 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:33:00,161 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 14:33:00,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 14:33:00.406 UTC [24699] WARNING: there is no transaction in progress
2026-08-10 14:33:00,414 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 14:33:00,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:33:00,659 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 14:33:00,659 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:33:00,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 14:33:00,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 14:33:01,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q9h9.c.gcp-upstream.internal','N') ;"
2026-08-10 14:33:01,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:33:01,644 [I] Table x_portal_user does not exist in database ranger
2026-08-10 14:33:01,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:33:01,891 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 14:33:01,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:33:02,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:33:02,392 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 14:33:02,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 14:33:03.239 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.254 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.269 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.282 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.296 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.312 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.369 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.375 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.388 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.398 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.407 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.420 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.428 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.434 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.440 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03.446 UTC [24913] WARNING: there is no transaction in progress
2026-08-10 14:33:03,971 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 14:33:03,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 14:33:04,194 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 14:33:04,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:33:04,443 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 14:33:04,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:33:04,685 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 14:33:04,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:33:04,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:33:05,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q9h9.c.gcp-upstream.internal';"
2026-08-10 14:33:05,395 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 14:33:05,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 14:33:05,647 [I] DB_PATCHES have already been applied
I20260810 14:33:05.653613 6331 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-admin:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task50NkZQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 14:33:05.653726 6331 mini_ranger.cc:193] Using host: 127.6.46.212
I20260810 14:33:05.674190 6331 mini_ranger.cc:227] Ranger admin URL: http://127.6.46.212:53935
Aug 10, 2026 2:33:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:33:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:33:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 2:33:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 2:33:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 2:33:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 2:33:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 14:33:06.924854 6331 mini_ranger.cc:160] Time spent starting Ranger: real 8.267s user 0.017s sys 0.038s
Aug 10, 2026 2:33:25 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 2:33:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 2:33:25 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 2:33:25 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 2:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 2:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 2:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 2:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 2:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 2:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 2:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 2:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 2:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 2:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 2:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 2:33:25 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 2:33:25 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 2:33:26 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 14:33:26.780824 6331 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/rangerkms_127.6.46.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/rangerkms_127.6.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.46.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.46.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.46.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/HTTP_127.6.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.46.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/HTTP_127.6.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 14:33:26 dist-test-slave-q9h9 krb5kdc[24511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372406, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 14:33:26.954272 6331 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 14:33:26.976557 6331 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 14:33:27.657775 6331 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 14:33:27.657917 6331 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-kms
I20260810 14:33:27.658080 6331 mini_ranger_kms.cc:227] Using postgres at 127.6.46.212:52809
2026-08-10 14:33:28,172 [I] DB FLAVOR :POSTGRES
2026-08-10 14:33:28,173 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:33:28,173 [I] Checking connection
2026-08-10 14:33:28,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 14:33:28,413 [I] connection success
2026-08-10 14:33:28,413 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 14:33:28,413 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 14:33:28,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 14:33:28,665 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 14:33:28,665 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:33:28,665 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 14:33:28,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:52809/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 14:33:28,909 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 14:33:29.360450 6331 mini_ranger_kms.cc:310] Created kms service
I20260810 14:33:29.529686 6331 mini_ranger_kms.cc:326] Created kudu user
I20260810 14:33:29.571614 6331 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 14:33:29 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372406, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:33:29.960457 6331 mini_ranger_kms.cc:384] Added ranger policy
I20260810 14:33:29.961769 6331 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-kms:/tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 14:33:29.961835 6331 mini_ranger_kms.cc:240] Using host: 127.6.46.212
I20260810 14:33:29.986446 6331 mini_ranger_kms.cc:276] Ranger KMS PID: 25214
I20260810 14:33:29.986657 6331 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.46.212:35659
Aug 10, 2026 2:33:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:33:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:33:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 2:33:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 2:33:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 2:33:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 14:33:32 dist-test-slave-q9h9 krb5kdc[24511](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372412, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.46.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:33:32.542362 6331 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.588s user 0.106s sys 0.447s
I20260810 14:33:32.542699 6331 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 14:33:32.542918 6331 mini_ranger_kms.cc:401] 127.6.46.212:35659/kms/v1/keys
Aug 10 14:33:32 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372412, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.46.212@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
Aug 10 14:33:33 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372412, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.46.212@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
Aug 10 14:33:35 dist-test-slave-q9h9 krb5kdc[24511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:33:35.387375 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:42123
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:42123
--ranger_config_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.46.212:35659/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:33:35.830492 25291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:33:35.830857 25291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:33:35.831090 25291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:33:35.839949 25291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:33:35.840062 25291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:33:35.840210 25291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:33:35.840387 25291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:33:35.840556 25291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:33:35.853709 25291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.46.212:35659/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:42123
--ranger_config_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:42123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:33:35.855311 25291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:33:35.856912 25291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:33:35.874599 25302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:33:35.874591 25299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:33:35.875022 25304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:33:35.876354 25291 server_base.cc:1034] running on GCE node
I20260810 14:33:35.877339 25291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:33:35.879314 25291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:33:35.880560 25291 hybrid_clock.cc:648] HybridClock initialized: now 1786372415880500 us; error 48 us; skew 500 ppm
I20260810 14:33:35.880960 25291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:33:35 dist-test-slave-q9h9 krb5kdc[24511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372415, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:33:35.884833 25291 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:33:35.886809 25291 webserver.cc:460] Webserver started at http://127.6.46.254:36737/ using document root <none> and password file <none>
I20260810 14:33:35.887595 25291 fs_manager.cc:359] Metadata directory not provided
I20260810 14:33:35.887715 25291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:33:35.888016 25291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:33:35 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372415, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:33:35.941987 25291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "890d404f7ac4444186d5853fe6dd100b"
format_stamp: "Formatted at 2026-08-10 14:33:35 on dist-test-slave-q9h9"
server_key: "2fe31079adbb9e1cb7ca66adf0f2f8ec"
server_key_iv: "1e1ad738be8be38371863f7804adea09"
server_key_version: "kuduclusterkey@0"
I20260810 14:33:35.943292 25291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "890d404f7ac4444186d5853fe6dd100b"
format_stamp: "Formatted at 2026-08-10 14:33:35 on dist-test-slave-q9h9"
server_key: "2fe31079adbb9e1cb7ca66adf0f2f8ec"
server_key_iv: "1e1ad738be8be38371863f7804adea09"
server_key_version: "kuduclusterkey@0"
I20260810 14:33:35.949931 25291 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 14:33:35.971692 25318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:33:35.973202 25291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:33:35.973377 25291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "890d404f7ac4444186d5853fe6dd100b"
format_stamp: "Formatted at 2026-08-10 14:33:35 on dist-test-slave-q9h9"
server_key: "2fe31079adbb9e1cb7ca66adf0f2f8ec"
server_key_iv: "1e1ad738be8be38371863f7804adea09"
server_key_version: "kuduclusterkey@0"
I20260810 14:33:35.973579 25291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:33:35.988879 25291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:33:35.993746 25291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:33:35.994345 25291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:33:36.028355 25291 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:42123
I20260810 14:33:36.028451 25391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:42123 every 8 connection(s)
I20260810 14:33:36.029821 25291 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:33:36.034811 25392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:33:36.037177 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 25291
I20260810 14:33:36.037403 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:33:36.038276 6331 external_mini_cluster.cc:1428] Setting key 05c93a538791b4369de04c87dad8d2c6
I20260810 14:33:36.050295 25392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b: Bootstrap starting.
I20260810 14:33:36.054611 25392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b: Neither blocks nor log segments found. Creating new log.
I20260810 14:33:36.055960 25392 log.cc:824] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:33:36.059535 25392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b: No bootstrap required, opened a new log
Aug 10 14:33:36 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:33:36.071197 25392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d404f7ac4444186d5853fe6dd100b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 42123 } }
I20260810 14:33:36.071620 25392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:33:36.071730 25392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890d404f7ac4444186d5853fe6dd100b, State: Initialized, Role: FOLLOWER
I20260810 14:33:36.072415 25392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d404f7ac4444186d5853fe6dd100b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 42123 } }
I20260810 14:33:36.072701 25392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:33:36.072840 25392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:33:36.073037 25392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:33:36.076779 25392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d404f7ac4444186d5853fe6dd100b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 42123 } }
I20260810 14:33:36.077567 25392 leader_election.cc:302] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 890d404f7ac4444186d5853fe6dd100b; no voters:
I20260810 14:33:36.078871 25392 leader_election.cc:288] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:33:36.079022 25402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:33:36.080881 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:36.040383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.6.46.254:42123)
0810 14:33:36.055216 (+ 14833us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:36.055231 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:36.055284 (+ 53us) server_negotiation.cc:407] Connection header received
0810 14:33:36.057949 (+ 2665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:36.057979 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:36.058459 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:36.058861 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:33:36.060409 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:36.062519 (+ 2110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:36.064408 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:36.065458 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:36.065731 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:36.070766 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:33:36.070777 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:33:36.070797 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:33:36.070837 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:33:36.074898 (+ 4061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:36.075720 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:36.075728 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:36.075735 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:36.076056 (+ 321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:36.076457 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:36.076462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:36.076465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:36.077271 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:33:36.077636 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:36.078900 (+ 1264us) server_negotiation.cc:299] Negotiation successful
0810 14:33:36.079343 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14569,"thread_start_us":14711,"threads_started":1}
I20260810 14:33:36.085677 25402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [term 1 LEADER]: Becoming Leader. State: Replica: 890d404f7ac4444186d5853fe6dd100b, State: Running, Role: LEADER
I20260810 14:33:36.087247 25392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:33:36.087241 25402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d404f7ac4444186d5853fe6dd100b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 42123 } }
I20260810 14:33:36.092494 25404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 890d404f7ac4444186d5853fe6dd100b. Latest consensus state: current_term: 1 leader_uuid: "890d404f7ac4444186d5853fe6dd100b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d404f7ac4444186d5853fe6dd100b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 42123 } } }
I20260810 14:33:36.092989 25404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [sys.catalog]: This master's current role is: LEADER
I20260810 14:33:36.092288 25403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "890d404f7ac4444186d5853fe6dd100b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890d404f7ac4444186d5853fe6dd100b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 42123 } } }
I20260810 14:33:36.096239 25403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b [sys.catalog]: This master's current role is: LEADER
I20260810 14:33:36.101269 25411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:33:36.108207 25411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:33:36.134954 25411 catalog_manager.cc:1347] Generated new cluster ID: 811aed07bcfd4448806be0bf9145b81c
I20260810 14:33:36.135149 25411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:33:36.135438 25392 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 14:33:36.188391 25411 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:33:36.190639 25411 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:33:36.213887 25411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b: Generated new TSK 0
I20260810 14:33:36.215188 25411 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:33:36.977466 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.46.212:35659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:42123
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:33:37 dist-test-slave-q9h9 krb5kdc[24511](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 14:33:37.457152 25470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:33:37.457599 25470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:33:37.457839 25470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:33:37.469215 25470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:33:37.469336 25470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:33:37.469565 25470 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:33:37.481564 25470 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.46.212:35659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:33:37.483053 25470 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:33:37.484354 25470 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:33:37.499063 25488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:33:37.499287 25489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:33:37.499783 25493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:33:37.500829 25470 server_base.cc:1034] running on GCE node
I20260810 14:33:37.501411 25470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:33:37.502393 25470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:33:37.503587 25470 hybrid_clock.cc:648] HybridClock initialized: now 1786372417503520 us; error 65 us; skew 500 ppm
I20260810 14:33:37.503885 25470 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:33:37 dist-test-slave-q9h9 krb5kdc[24511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:33:37.507010 25470 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:33:37.508683 25470 webserver.cc:460] Webserver started at http://127.6.46.193:44039/ using document root <none> and password file <none>
I20260810 14:33:37.509532 25470 fs_manager.cc:359] Metadata directory not provided
I20260810 14:33:37.509789 25470 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:33:37.510245 25470 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:33:37 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:33:37.535084 25470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "1e2cf075c9fe4883a780f053026f8833"
format_stamp: "Formatted at 2026-08-10 14:33:37 on dist-test-slave-q9h9"
server_key: "9fb3f56db54abc79b273725273257da8"
server_key_iv: "f1d2e34217cb27343848ebb4562ac754"
server_key_version: "kuduclusterkey@0"
I20260810 14:33:37.536339 25470 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "1e2cf075c9fe4883a780f053026f8833"
format_stamp: "Formatted at 2026-08-10 14:33:37 on dist-test-slave-q9h9"
server_key: "9fb3f56db54abc79b273725273257da8"
server_key_iv: "f1d2e34217cb27343848ebb4562ac754"
server_key_version: "kuduclusterkey@0"
I20260810 14:33:37.550031 25470 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.008s
I20260810 14:33:37.586885 25502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:33:37.589104 25470 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.010s sys 0.006s
I20260810 14:33:37.589370 25470 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "1e2cf075c9fe4883a780f053026f8833"
format_stamp: "Formatted at 2026-08-10 14:33:37 on dist-test-slave-q9h9"
server_key: "9fb3f56db54abc79b273725273257da8"
server_key_iv: "f1d2e34217cb27343848ebb4562ac754"
server_key_version: "kuduclusterkey@0"
I20260810 14:33:37.589743 25470 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:33:37.667464 25470 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:33:37.673369 25470 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:33:37.673810 25470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:33:37.681351 25470 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:33:37.699342 25470 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:33:37.699569 25470 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 14:33:37.699895 25470 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:33:37.700119 25470 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:33:37 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:33:38.032246 25470 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:33153
I20260810 14:33:38.032232 25679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:33153 every 8 connection(s)
I20260810 14:33:38.034335 25470 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:33:38.042300 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 25470
I20260810 14:33:38.042501 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:33:38.043265 6331 external_mini_cluster.cc:1428] Setting key b599df479f60965398595878590f5782
Aug 10 14:33:38 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:33:38.093830 25684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:38.041436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:42123 (local address 127.6.46.193:39349)
0810 14:33:38.071607 (+ 30171us) negotiation.cc:107] Waiting for socket to connect
0810 14:33:38.071671 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 14:33:38.074890 (+ 3219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:33:38.076051 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:33:38.076066 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:33:38.076791 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:33:38.077862 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:38.077891 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:38.079277 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:38.079280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:38.080483 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:38.080495 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:38.081636 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:38.081644 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:38.081911 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:38.082678 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:33:38.082706 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:33:38.085518 (+ 2812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:33:38.089434 (+ 3916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:33:38.089443 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:33:38.089460 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:33:38.090005 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:33:38.090424 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:33:38.090428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:33:38.090431 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:33:38.090667 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:33:38.091351 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:33:38.091358 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:33:38.091751 (+ 393us) client_negotiation.cc:769] Sending connection context
0810 14:33:38.092018 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 14:33:38.092482 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29792,"thread_start_us":29534,"threads_started":1}
I20260810 14:33:38.095038 25683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:38.040676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:39349 (local address 127.6.46.254:42123)
0810 14:33:38.074484 (+ 33808us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:38.074492 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:38.074518 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:33:38.075260 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:38.075267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:38.075357 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:38.075442 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:33:38.078109 (+ 2667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:38.079119 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:38.080632 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:38.081217 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:38.081333 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:38.085762 (+ 4429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:33:38.085768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:33:38.085773 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:33:38.085810 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:33:38.089244 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:38.090158 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:38.090163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:38.090166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:38.090307 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:38.090797 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:38.090802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:38.090805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:38.091154 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:33:38.091279 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:38.094446 (+ 3167us) server_negotiation.cc:299] Negotiation successful
0810 14:33:38.094679 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33657,"thread_start_us":33479,"threads_started":1}
I20260810 14:33:38.097322 25680 heartbeater.cc:347] Connected to a master server at 127.6.46.254:42123
I20260810 14:33:38.097666 25680 heartbeater.cc:464] Registering TS with master...
I20260810 14:33:38.098549 25680 heartbeater.cc:511] Master 127.6.46.254:42123 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
I20260810 14:33:38.101157 25340 ts_manager.cc:194] Registered new tserver with Master: 1e2cf075c9fe4883a780f053026f8833 (127.6.46.193:33153)
I20260810 14:33:38.102905 25340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:39349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:33:38.138731 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.46.212:35659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:42123
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:33:38 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372417, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
W20260810 14:33:38.648933 25690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:33:38.649295 25690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:33:38.649544 25690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:33:38.657236 25690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:33:38.657382 25690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:33:38.657548 25690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:33:38.667501 25690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.46.212:35659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:33:38.668711 25690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:33:38.670225 25690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:33:38.684240 25715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:33:38.684183 25720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:33:38.684161 25716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:33:38.685032 25690 server_base.cc:1034] running on GCE node
I20260810 14:33:38.685562 25690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:33:38.686431 25690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:33:38.687652 25690 hybrid_clock.cc:648] HybridClock initialized: now 1786372418687592 us; error 49 us; skew 500 ppm
I20260810 14:33:38.687945 25690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:33:38 dist-test-slave-q9h9 krb5kdc[24511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:33:38.691004 25690 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:33:38.692435 25690 webserver.cc:460] Webserver started at http://127.6.46.194:37447/ using document root <none> and password file <none>
I20260810 14:33:38.693063 25690 fs_manager.cc:359] Metadata directory not provided
I20260810 14:33:38.693236 25690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:33:38.693557 25690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:33:38 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:33:38.710320 25690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "80d7dd34399646e697152af5fc032dda"
format_stamp: "Formatted at 2026-08-10 14:33:38 on dist-test-slave-q9h9"
server_key: "0286cb3944e165617edf02c701ebec91"
server_key_iv: "c5d5eda4b3762f720cd5bef323d6e99a"
server_key_version: "kuduclusterkey@0"
I20260810 14:33:38.711233 25690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "80d7dd34399646e697152af5fc032dda"
format_stamp: "Formatted at 2026-08-10 14:33:38 on dist-test-slave-q9h9"
server_key: "0286cb3944e165617edf02c701ebec91"
server_key_iv: "c5d5eda4b3762f720cd5bef323d6e99a"
server_key_version: "kuduclusterkey@0"
I20260810 14:33:38.716626 25690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 14:33:38.733414 25729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:33:38.735061 25690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 14:33:38.735245 25690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "80d7dd34399646e697152af5fc032dda"
format_stamp: "Formatted at 2026-08-10 14:33:38 on dist-test-slave-q9h9"
server_key: "0286cb3944e165617edf02c701ebec91"
server_key_iv: "c5d5eda4b3762f720cd5bef323d6e99a"
server_key_version: "kuduclusterkey@0"
I20260810 14:33:38.735474 25690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:33:38.757375 25690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:33:38.762513 25690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:33:38.763149 25690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:33:38.764474 25690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:33:38.766295 25690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:33:38.766392 25690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:33:38.766526 25690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:33:38.766595 25690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:33:38.813786 25690 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:33795
I20260810 14:33:38.813855 25921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:33795 every 8 connection(s)
I20260810 14:33:38.815059 25690 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:33:38.819523 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 25690
I20260810 14:33:38.819710 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:33:38.820282 6331 external_mini_cluster.cc:1428] Setting key 28ace1136ecb4f4b54f528ed2bc1c6bb
Aug 10 14:33:38 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372418, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:33:38.835187 25929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:38.817619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:43115 (local address 127.6.46.254:42123)
0810 14:33:38.818116 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:38.818123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:38.819191 (+ 1068us) server_negotiation.cc:407] Connection header received
0810 14:33:38.821478 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:38.821484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:38.821563 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:38.821655 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:33:38.823447 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:38.824179 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:38.825767 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:38.826219 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:38.826333 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:38.829393 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:33:38.829398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:33:38.829402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:33:38.829435 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:33:38.831837 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:38.832531 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:38.832535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:38.832538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:38.832705 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:38.833122 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:38.833127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:38.833129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:38.833462 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:33:38.833620 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:38.834428 (+ 808us) server_negotiation.cc:299] Negotiation successful
0810 14:33:38.834636 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":212,"threads_started":1}
I20260810 14:33:38.835522 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:38.818216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:42123 (local address 127.6.46.194:43115)
0810 14:33:38.819021 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0810 14:33:38.819071 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 14:33:38.821203 (+ 2132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:33:38.821910 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:33:38.821921 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:33:38.822483 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:33:38.823279 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:38.823303 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:38.824393 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:38.824397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:38.825623 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:38.825636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:38.826438 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:38.826447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:38.826659 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:38.827254 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:33:38.827276 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:33:38.829174 (+ 1898us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:33:38.831966 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:33:38.831975 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:33:38.831990 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:33:38.832415 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:33:38.832825 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:33:38.832828 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:33:38.832831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:33:38.833023 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:33:38.833605 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:33:38.833612 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:33:38.833942 (+ 330us) client_negotiation.cc:769] Sending connection context
0810 14:33:38.834218 (+ 276us) client_negotiation.cc:240] Negotiation successful
0810 14:33:38.834550 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":322,"threads_started":1}
I20260810 14:33:38.836850 25927 heartbeater.cc:347] Connected to a master server at 127.6.46.254:42123
I20260810 14:33:38.837110 25927 heartbeater.cc:464] Registering TS with master...
I20260810 14:33:38.837766 25927 heartbeater.cc:511] Master 127.6.46.254:42123 requested a full tablet report, sending...
I20260810 14:33:38.839248 25340 ts_manager.cc:194] Registered new tserver with Master: 80d7dd34399646e697152af5fc032dda (127.6.46.194:33795)
I20260810 14:33:38.840018 25340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:43115
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:33:38.874341 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.46.212:35659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:42123
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:33:39.106249 25680 heartbeater.cc:503] Master 127.6.46.254:42123 was elected leader, sending a full tablet report...
W20260810 14:33:39.277051 25934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:33:39.277365 25934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:33:39.277510 25934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:33:39.286335 25934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:33:39.286445 25934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:33:39.286592 25934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:33:39.296890 25934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.46.212:35659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:42123
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:33:39.298231 25934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:33:39.299480 25934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:33:39.314159 25940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:33:39.314091 25946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:33:39.314481 25942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:33:39.315115 25934 server_base.cc:1034] running on GCE node
I20260810 14:33:39.315675 25934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:33:39.316551 25934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:33:39.317757 25934 hybrid_clock.cc:648] HybridClock initialized: now 1786372419317704 us; error 48 us; skew 500 ppm
I20260810 14:33:39.318070 25934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:33:39 dist-test-slave-q9h9 krb5kdc[24511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372419, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:33:39.321698 25934 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:33:39.323153 25934 webserver.cc:460] Webserver started at http://127.6.46.195:33739/ using document root <none> and password file <none>
I20260810 14:33:39.323779 25934 fs_manager.cc:359] Metadata directory not provided
I20260810 14:33:39.323899 25934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:33:39.324188 25934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:33:39 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372419, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:33:39.339921 25934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "8a92a790260f4f9e998638acef7a3308"
format_stamp: "Formatted at 2026-08-10 14:33:39 on dist-test-slave-q9h9"
server_key: "b97dae662391461ec031d3944c5bffde"
server_key_iv: "a360a09bde2eafdfdc05992188172c9a"
server_key_version: "kuduclusterkey@0"
I20260810 14:33:39.340855 25934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "8a92a790260f4f9e998638acef7a3308"
format_stamp: "Formatted at 2026-08-10 14:33:39 on dist-test-slave-q9h9"
server_key: "b97dae662391461ec031d3944c5bffde"
server_key_iv: "a360a09bde2eafdfdc05992188172c9a"
server_key_version: "kuduclusterkey@0"
I20260810 14:33:39.346529 25934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 14:33:39.362852 25959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:33:39.365228 25934 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 14:33:39.365470 25934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "8a92a790260f4f9e998638acef7a3308"
format_stamp: "Formatted at 2026-08-10 14:33:39 on dist-test-slave-q9h9"
server_key: "b97dae662391461ec031d3944c5bffde"
server_key_iv: "a360a09bde2eafdfdc05992188172c9a"
server_key_version: "kuduclusterkey@0"
I20260810 14:33:39.365693 25934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:33:39.384934 25934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:33:39.389271 25934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:33:39.389547 25934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:33:39.390617 25934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:33:39.392591 25934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:33:39.392669 25934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:33:39.392830 25934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:33:39.392920 25934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:33:39.445080 25934 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:35987
I20260810 14:33:39.445067 26144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:35987 every 8 connection(s)
I20260810 14:33:39.446396 25934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:33:39.446772 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 25934
I20260810 14:33:39.446925 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:33:39.447386 6331 external_mini_cluster.cc:1428] Setting key 9357844c09bb6c34ea1bf9be6671d5f4
Aug 10 14:33:39 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372419, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:33:39.467437 26148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.449638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:46991 (local address 127.6.46.254:42123)
0810 14:33:39.450279 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:39.450285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:39.451495 (+ 1210us) server_negotiation.cc:407] Connection header received
0810 14:33:39.453704 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:39.453708 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:39.453785 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:39.453875 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:33:39.455382 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.456081 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.457629 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.458072 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.458169 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.461295 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:33:39.461300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:33:39.461303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:33:39.461332 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:33:39.464673 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:39.465292 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:39.465296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:39.465298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:39.465461 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:39.465936 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:39.465941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:39.465943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:39.466322 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:33:39.466450 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:39.467021 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 14:33:39.467182 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":369,"threads_started":1}
I20260810 14:33:39.468475 26149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.450315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:42123 (local address 127.6.46.195:46991)
0810 14:33:39.451337 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0810 14:33:39.451383 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:33:39.453456 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:33:39.454139 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:33:39.454147 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:33:39.454574 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:33:39.455226 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:39.455247 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.456244 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.456248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:39.457482 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:39.457503 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.458239 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.458243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:39.458433 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.459021 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:33:39.459043 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:33:39.461099 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:33:39.464788 (+ 3689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:33:39.464795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:33:39.464805 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:33:39.465208 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:33:39.465539 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:33:39.465542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:33:39.465545 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:33:39.465809 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:33:39.466429 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:33:39.466435 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:33:39.466691 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 14:33:39.466956 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 14:33:39.467390 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":482,"threads_started":1}
I20260810 14:33:39.469638 26145 heartbeater.cc:347] Connected to a master server at 127.6.46.254:42123
I20260810 14:33:39.469964 26145 heartbeater.cc:464] Registering TS with master...
I20260810 14:33:39.470959 26145 heartbeater.cc:511] Master 127.6.46.254:42123 requested a full tablet report, sending...
I20260810 14:33:39.472267 25340 ts_manager.cc:194] Registered new tserver with Master: 8a92a790260f4f9e998638acef7a3308 (127.6.46.195:35987)
I20260810 14:33:39.473235 25340 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:46991
I20260810 14:33:39.475582 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:33:39.493883 26148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.481184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.6.46.254:42123)
0810 14:33:39.481375 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:39.481381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:39.481819 (+ 438us) server_negotiation.cc:407] Connection header received
0810 14:33:39.481942 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:39.481945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:39.482029 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:39.482108 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:33:39.483477 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.484241 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.485576 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.485977 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.486116 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.488412 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:33:39.488417 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:33:39.488420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:33:39.488448 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:33:39.491265 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:39.491909 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:39.491913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:39.491915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:39.492084 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:39.492479 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:39.492483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:39.492490 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:39.492796 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:33:39.493087 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:39.493393 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 14:33:39.493615 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:33:39.498121 25340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:33:39.505681 25340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:33:39.531584 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.520327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.6.46.193:33153)
0810 14:33:39.521582 (+ 1255us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:39.521592 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:39.522588 (+ 996us) server_negotiation.cc:407] Connection header received
0810 14:33:39.522931 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:39.522939 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:39.523179 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:39.523343 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:33:39.524721 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.526258 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.529231 (+ 2973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.530421 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.530560 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.530791 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:39.531048 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 14:33:39.531207 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1038,"thread_start_us":742,"threads_started":1}
I20260810 14:33:39.531570 26171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.520008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:33153 (local address 127.0.0.1:40770)
0810 14:33:39.522446 (+ 2438us) negotiation.cc:107] Waiting for socket to connect
0810 14:33:39.522474 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 14:33:39.522663 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:33:39.523691 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:33:39.523699 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:33:39.523733 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:33:39.524400 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:39.524413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.526515 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.526521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:39.529090 (+ 2569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:39.529103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.530635 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.530641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:39.530818 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.530871 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 14:33:39.530977 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 14:33:39.531155 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2231,"mutex_wait_us":6,"spinlock_wait_cycles":22528,"thread_start_us":2693,"threads_started":1}
I20260810 14:33:39.533720 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.522926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:33795 (local address 127.0.0.1:47516)
0810 14:33:39.524082 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0810 14:33:39.524101 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:33:39.524215 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:33:39.525903 (+ 1688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:33:39.525908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:33:39.525932 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:33:39.526532 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:39.526545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.527979 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.527989 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:39.530653 (+ 2664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:39.530666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.532908 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.532917 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:39.533090 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.533129 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 14:33:39.533234 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 14:33:39.533351 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"mutex_wait_us":184,"thread_start_us":774,"threads_started":1}
I20260810 14:33:39.533825 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.523427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47516 (local address 127.6.46.194:33795)
0810 14:33:39.525104 (+ 1677us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:39.525112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:39.525141 (+ 29us) server_negotiation.cc:407] Connection header received
0810 14:33:39.525248 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:39.525255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:39.525519 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:39.525686 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:33:39.526803 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.527826 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.530818 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.532279 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.532400 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.532562 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:39.533298 (+ 736us) server_negotiation.cc:299] Negotiation successful
0810 14:33:39.533467 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1518,"thread_start_us":1400,"threads_started":1}
I20260810 14:33:39.534221 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.520098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50910 (local address 127.6.46.195:35987)
0810 14:33:39.522070 (+ 1972us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:39.522079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:39.522680 (+ 601us) server_negotiation.cc:407] Connection header received
0810 14:33:39.522969 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:39.522976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:39.523178 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:39.523345 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:33:39.524664 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.526440 (+ 1776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.530701 (+ 4261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.532193 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.532303 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.532562 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:39.533461 (+ 899us) server_negotiation.cc:299] Negotiation successful
0810 14:33:39.533693 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1770,"thread_start_us":1509,"threads_started":1}
I20260810 14:33:39.534914 26173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.519769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:35987 (local address 127.0.0.1:50910)
0810 14:33:39.522457 (+ 2688us) negotiation.cc:107] Waiting for socket to connect
0810 14:33:39.522478 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:33:39.522660 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:33:39.523851 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:33:39.523856 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:33:39.523885 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:33:39.524452 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:39.524465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.526600 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.526605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:39.530498 (+ 3893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:39.530513 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.533021 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.533029 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:39.533272 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.533302 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 14:33:39.534405 (+ 1103us) client_negotiation.cc:240] Negotiation successful
0810 14:33:39.534520 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2171,"mutex_wait_us":77,"spinlock_wait_cycles":4480,"thread_start_us":1920,"threads_started":1}
I20260810 14:33:39.536046 25576 tablet_service.cc:1467] Processing CreateTablet for tablet 63494241ffdc47e2bc9d77699fb8a68f (DEFAULT_TABLE table=test-table [id=4a67cf1b878849438d7b67d6ad5c7147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:33:39.537271 25576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63494241ffdc47e2bc9d77699fb8a68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:33:39.541574 25806 tablet_service.cc:1467] Processing CreateTablet for tablet 63494241ffdc47e2bc9d77699fb8a68f (DEFAULT_TABLE table=test-table [id=4a67cf1b878849438d7b67d6ad5c7147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:33:39.541692 26045 tablet_service.cc:1467] Processing CreateTablet for tablet 63494241ffdc47e2bc9d77699fb8a68f (DEFAULT_TABLE table=test-table [id=4a67cf1b878849438d7b67d6ad5c7147]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:33:39.543047 25806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63494241ffdc47e2bc9d77699fb8a68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:33:39.543136 26045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63494241ffdc47e2bc9d77699fb8a68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:33:39.549484 26184 tablet_bootstrap.cc:492] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833: Bootstrap starting.
I20260810 14:33:39.553455 26184 tablet_bootstrap.cc:654] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833: Neither blocks nor log segments found. Creating new log.
I20260810 14:33:39.554986 26184 log.cc:824] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833: Log is configured to *not* fsync() on all Append() calls
I20260810 14:33:39.558485 26188 tablet_bootstrap.cc:492] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308: Bootstrap starting.
I20260810 14:33:39.559113 26184 tablet_bootstrap.cc:492] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833: No bootstrap required, opened a new log
I20260810 14:33:39.559450 26184 ts_tablet_manager.cc:1397] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260810 14:33:39.562584 26188 tablet_bootstrap.cc:654] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308: Neither blocks nor log segments found. Creating new log.
I20260810 14:33:39.562845 26189 tablet_bootstrap.cc:492] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda: Bootstrap starting.
I20260810 14:33:39.564445 26188 log.cc:824] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308: Log is configured to *not* fsync() on all Append() calls
I20260810 14:33:39.567191 26189 tablet_bootstrap.cc:654] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda: Neither blocks nor log segments found. Creating new log.
I20260810 14:33:39.567910 26188 tablet_bootstrap.cc:492] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308: No bootstrap required, opened a new log
I20260810 14:33:39.568177 26188 ts_tablet_manager.cc:1397] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 14:33:39.568560 26189 log.cc:824] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda: Log is configured to *not* fsync() on all Append() calls
I20260810 14:33:39.572165 26189 tablet_bootstrap.cc:492] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda: No bootstrap required, opened a new log
I20260810 14:33:39.571935 26184 raft_consensus.cc:348] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92a790260f4f9e998638acef7a3308" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35987 } } peers { permanent_uuid: "1e2cf075c9fe4883a780f053026f8833" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33153 } } peers { permanent_uuid: "80d7dd34399646e697152af5fc032dda" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 33795 } }
I20260810 14:33:39.572407 26189 ts_tablet_manager.cc:1397] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 14:33:39.572420 26184 raft_consensus.cc:374] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:33:39.572520 26184 raft_consensus.cc:729] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e2cf075c9fe4883a780f053026f8833, State: Initialized, Role: FOLLOWER
I20260810 14:33:39.573175 26184 consensus_queue.cc:260] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92a790260f4f9e998638acef7a3308" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35987 } } peers { permanent_uuid: "1e2cf075c9fe4883a780f053026f8833" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33153 } } peers { permanent_uuid: "80d7dd34399646e697152af5fc032dda" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 33795 } }
I20260810 14:33:39.576526 26184 ts_tablet_manager.cc:1428] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
W20260810 14:33:39.580250 25681 tablet.cc:2369] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:33:39.580746 26188 raft_consensus.cc:348] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92a790260f4f9e998638acef7a3308" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35987 } } peers { permanent_uuid: "1e2cf075c9fe4883a780f053026f8833" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33153 } } peers { permanent_uuid: "80d7dd34399646e697152af5fc032dda" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 33795 } }
I20260810 14:33:39.581434 26188 raft_consensus.cc:374] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:33:39.581625 26188 raft_consensus.cc:729] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a92a790260f4f9e998638acef7a3308, State: Initialized, Role: FOLLOWER
I20260810 14:33:39.582415 26188 consensus_queue.cc:260] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92a790260f4f9e998638acef7a3308" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35987 } } peers { permanent_uuid: "1e2cf075c9fe4883a780f053026f8833" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33153 } } peers { permanent_uuid: "80d7dd34399646e697152af5fc032dda" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 33795 } }
I20260810 14:33:39.582785 26189 raft_consensus.cc:348] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92a790260f4f9e998638acef7a3308" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35987 } } peers { permanent_uuid: "1e2cf075c9fe4883a780f053026f8833" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33153 } } peers { permanent_uuid: "80d7dd34399646e697152af5fc032dda" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 33795 } }
I20260810 14:33:39.583089 26189 raft_consensus.cc:374] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:33:39.583192 26189 raft_consensus.cc:729] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80d7dd34399646e697152af5fc032dda, State: Initialized, Role: FOLLOWER
I20260810 14:33:39.583624 26145 heartbeater.cc:503] Master 127.6.46.254:42123 was elected leader, sending a full tablet report...
I20260810 14:33:39.583678 26189 consensus_queue.cc:260] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92a790260f4f9e998638acef7a3308" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35987 } } peers { permanent_uuid: "1e2cf075c9fe4883a780f053026f8833" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33153 } } peers { permanent_uuid: "80d7dd34399646e697152af5fc032dda" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 33795 } }
I20260810 14:33:39.584337 26188 ts_tablet_manager.cc:1428] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260810 14:33:39.584851 25927 heartbeater.cc:503] Master 127.6.46.254:42123 was elected leader, sending a full tablet report...
I20260810 14:33:39.585354 26189 ts_tablet_manager.cc:1428] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
W20260810 14:33:39.698518 26147 tablet.cc:2369] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:33:39.817454 25928 tablet.cc:2369] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:33:39.886384 26195 raft_consensus.cc:482] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:33:39.886883 26195 raft_consensus.cc:504] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92a790260f4f9e998638acef7a3308" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35987 } } peers { permanent_uuid: "1e2cf075c9fe4883a780f053026f8833" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33153 } } peers { permanent_uuid: "80d7dd34399646e697152af5fc032dda" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 33795 } }
I20260810 14:33:39.888926 26195 leader_election.cc:288] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a92a790260f4f9e998638acef7a3308 (127.6.46.195:35987), 80d7dd34399646e697152af5fc032dda (127.6.46.194:33795)
I20260810 14:33:39.897051 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.889249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:58579 (local address 127.6.46.195:35987)
0810 14:33:39.889419 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:39.889423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:39.890149 (+ 726us) server_negotiation.cc:407] Connection header received
0810 14:33:39.890394 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:39.890400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:39.890492 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:39.890586 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:33:39.891453 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.892307 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.894808 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.895997 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.896161 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.896233 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:39.896667 (+ 434us) server_negotiation.cc:299] Negotiation successful
0810 14:33:39.896802 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:33:39.897130 26199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.889242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:35987 (local address 127.6.46.193:58579)
0810 14:33:39.890012 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 14:33:39.890060 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:33:39.890210 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:33:39.890740 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:33:39.890745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:33:39.890778 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:33:39.891304 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:39.891318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.892442 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.892448 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:39.894648 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:39.894669 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.896270 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.896277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:39.896488 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.896526 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 14:33:39.896604 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:33:39.896693 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":363,"threads_started":1}
I20260810 14:33:39.898063 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.890438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:59653 (local address 127.6.46.194:33795)
0810 14:33:39.890639 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:39.890644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:39.891365 (+ 721us) server_negotiation.cc:407] Connection header received
0810 14:33:39.891578 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:39.891582 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:39.891671 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:39.891793 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:33:39.892605 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.893357 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.895742 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.897137 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.897331 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.897404 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:39.897763 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 14:33:39.897859 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:33:39.898309 26200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.890217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:33795 (local address 127.6.46.193:59653)
0810 14:33:39.891257 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0810 14:33:39.891277 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:33:39.891402 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:33:39.891809 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:33:39.891814 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:33:39.891840 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:33:39.892431 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:39.892442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.893510 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.893516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:39.895568 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:33:39.895581 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.897420 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.897427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:33:39.897563 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.897593 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 14:33:39.897705 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 14:33:39.897831 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"thread_start_us":644,"threads_started":1}
I20260810 14:33:39.898418 26077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63494241ffdc47e2bc9d77699fb8a68f" candidate_uuid: "1e2cf075c9fe4883a780f053026f8833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a92a790260f4f9e998638acef7a3308" is_pre_election: true
I20260810 14:33:39.899012 26077 raft_consensus.cc:2454] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2cf075c9fe4883a780f053026f8833 in term 0.
I20260810 14:33:39.899457 25842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63494241ffdc47e2bc9d77699fb8a68f" candidate_uuid: "1e2cf075c9fe4883a780f053026f8833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d7dd34399646e697152af5fc032dda" is_pre_election: true
I20260810 14:33:39.899685 25505 leader_election.cc:302] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2cf075c9fe4883a780f053026f8833, 8a92a790260f4f9e998638acef7a3308; no voters:
I20260810 14:33:39.899922 25842 raft_consensus.cc:2454] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e2cf075c9fe4883a780f053026f8833 in term 0.
I20260810 14:33:39.900113 26195 raft_consensus.cc:2781] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:33:39.900269 26195 raft_consensus.cc:482] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:33:39.900377 26195 raft_consensus.cc:3037] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:33:39.903038 26195 raft_consensus.cc:504] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92a790260f4f9e998638acef7a3308" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35987 } } peers { permanent_uuid: "1e2cf075c9fe4883a780f053026f8833" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33153 } } peers { permanent_uuid: "80d7dd34399646e697152af5fc032dda" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 33795 } }
I20260810 14:33:39.903864 26195 leader_election.cc:288] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [CANDIDATE]: Term 1 election: Requested vote from peers 8a92a790260f4f9e998638acef7a3308 (127.6.46.195:35987), 80d7dd34399646e697152af5fc032dda (127.6.46.194:33795)
I20260810 14:33:39.904189 26077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63494241ffdc47e2bc9d77699fb8a68f" candidate_uuid: "1e2cf075c9fe4883a780f053026f8833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a92a790260f4f9e998638acef7a3308"
I20260810 14:33:39.904412 26077 raft_consensus.cc:3037] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:33:39.904377 25842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63494241ffdc47e2bc9d77699fb8a68f" candidate_uuid: "1e2cf075c9fe4883a780f053026f8833" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "80d7dd34399646e697152af5fc032dda"
I20260810 14:33:39.904587 25842 raft_consensus.cc:3037] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:33:39.908176 26077 raft_consensus.cc:2454] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2cf075c9fe4883a780f053026f8833 in term 1.
I20260810 14:33:39.908200 25842 raft_consensus.cc:2454] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e2cf075c9fe4883a780f053026f8833 in term 1.
I20260810 14:33:39.908686 25505 leader_election.cc:302] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e2cf075c9fe4883a780f053026f8833, 80d7dd34399646e697152af5fc032dda; no voters:
I20260810 14:33:39.908998 26195 raft_consensus.cc:2781] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:33:39.909561 26195 raft_consensus.cc:686] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [term 1 LEADER]: Becoming Leader. State: Replica: 1e2cf075c9fe4883a780f053026f8833, State: Running, Role: LEADER
I20260810 14:33:39.910106 26195 consensus_queue.cc:237] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92a790260f4f9e998638acef7a3308" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35987 } } peers { permanent_uuid: "1e2cf075c9fe4883a780f053026f8833" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33153 } } peers { permanent_uuid: "80d7dd34399646e697152af5fc032dda" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 33795 } }
I20260810 14:33:39.914700 25339 catalog_manager.cc:5495] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e2cf075c9fe4883a780f053026f8833 (127.6.46.193). New cstate: current_term: 1 leader_uuid: "1e2cf075c9fe4883a780f053026f8833" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a92a790260f4f9e998638acef7a3308" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e2cf075c9fe4883a780f053026f8833" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 33153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "80d7dd34399646e697152af5fc032dda" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 33795 } health_report { overall_health: UNKNOWN } } }
I20260810 14:33:39.988606 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:39.983304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40772 (local address 127.6.46.193:33153)
0810 14:33:39.983488 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:39.983492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:39.983507 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:33:39.983598 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:39.983603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:39.983699 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:39.983828 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:33:39.984200 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.984914 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.986455 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:39.986927 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:39.987039 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:39.987561 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:33:39.988012 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:33:39.988145 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:39.988276 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 14:33:39.988355 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:33:39.997193 25842 raft_consensus.cc:1261] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda [term 1 FOLLOWER]: Refusing update from remote peer 1e2cf075c9fe4883a780f053026f8833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:33:39.997349 26077 raft_consensus.cc:1261] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308 [term 1 FOLLOWER]: Refusing update from remote peer 1e2cf075c9fe4883a780f053026f8833: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:33:39.998188 26206 consensus_queue.cc:1035] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "80d7dd34399646e697152af5fc032dda" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 33795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:33:39.998515 26195 consensus_queue.cc:1035] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a92a790260f4f9e998638acef7a3308" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:33:40.014335 26212 mvcc.cc:205] Tried to move back new op lower bound from 7316981432298360832 to 7316981431955443712. Current Snapshot: MvccSnapshot[applied={T|T < 7316981432298360832}]
I20260810 14:33:40.029397 25339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:33:40.029670 25339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:33:40.033557 25339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 890d404f7ac4444186d5853fe6dd100b: Sending DeleteTablet for 3 replicas of tablet 63494241ffdc47e2bc9d77699fb8a68f
I20260810 14:33:40.035210 26045 tablet_service.cc:1514] Processing DeleteTablet for tablet 63494241ffdc47e2bc9d77699fb8a68f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:33:40 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:50910
I20260810 14:33:40.035842 25806 tablet_service.cc:1514] Processing DeleteTablet for tablet 63494241ffdc47e2bc9d77699fb8a68f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:33:40 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:47516
I20260810 14:33:40.037652 25576 tablet_service.cc:1514] Processing DeleteTablet for tablet 63494241ffdc47e2bc9d77699fb8a68f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:33:40 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:40770
I20260810 14:33:40.041268 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 25470
I20260810 14:33:40.051636 26227 tablet_replica.cc:330] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833: stopping tablet replica
I20260810 14:33:40.052033 26227 raft_consensus.cc:2229] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:33:40.052733 26227 raft_consensus.cc:2258] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:33:40.054754 26227 ts_tablet_manager.cc:1905] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:33:40.054780 26225 tablet_replica.cc:330] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308: stopping tablet replica
I20260810 14:33:40.055109 26226 tablet_replica.cc:330] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda: stopping tablet replica
I20260810 14:33:40.055164 26225 raft_consensus.cc:2229] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:33:40.055488 26226 raft_consensus.cc:2229] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:33:40.055969 26226 raft_consensus.cc:2258] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:33:40.057766 26226 ts_tablet_manager.cc:1905] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:33:40.057852 26225 raft_consensus.cc:2258] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:33:40.059839 26225 ts_tablet_manager.cc:1905] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:33:40.064352 26227 ts_tablet_manager.cc:1918] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:33:40.064471 26226 ts_tablet_manager.cc:1918] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:33:40.064630 26226 log.cc:1196] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/63494241ffdc47e2bc9d77699fb8a68f
I20260810 14:33:40.065155 26226 ts_tablet_manager.cc:1939] T 63494241ffdc47e2bc9d77699fb8a68f P 80d7dd34399646e697152af5fc032dda: Deleting consensus metadata
I20260810 14:33:40.066845 25321 catalog_manager.cc:4841] TS 80d7dd34399646e697152af5fc032dda (127.6.46.194:33795): tablet 63494241ffdc47e2bc9d77699fb8a68f (table test-table [id=4a67cf1b878849438d7b67d6ad5c7147]) successfully deleted
Aug 10 14:33:40 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:33:40.068819 26227 log.cc:1196] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/63494241ffdc47e2bc9d77699fb8a68f
I20260810 14:33:40.069631 26227 ts_tablet_manager.cc:1939] T 63494241ffdc47e2bc9d77699fb8a68f P 1e2cf075c9fe4883a780f053026f8833: Deleting consensus metadata
I20260810 14:33:40.071470 26225 ts_tablet_manager.cc:1918] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:33:40.071626 26225 log.cc:1196] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/63494241ffdc47e2bc9d77699fb8a68f
I20260810 14:33:40.072213 26225 ts_tablet_manager.cc:1939] T 63494241ffdc47e2bc9d77699fb8a68f P 8a92a790260f4f9e998638acef7a3308: Deleting consensus metadata
I20260810 14:33:40.073262 25324 catalog_manager.cc:4841] TS 1e2cf075c9fe4883a780f053026f8833 (127.6.46.193:33153): tablet 63494241ffdc47e2bc9d77699fb8a68f (table test-table [id=4a67cf1b878849438d7b67d6ad5c7147]) successfully deleted
I20260810 14:33:40.073900 25321 catalog_manager.cc:4841] TS 8a92a790260f4f9e998638acef7a3308 (127.6.46.195:35987): tablet 63494241ffdc47e2bc9d77699fb8a68f (table test-table [id=4a67cf1b878849438d7b67d6ad5c7147]) successfully deleted
I20260810 14:33:40.076114 26175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:40.042679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.6.46.193:33153)
0810 14:33:40.042830 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:40.042836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:40.054174 (+ 11338us) server_negotiation.cc:407] Connection header received
0810 14:33:40.055552 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:40.055558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:40.055645 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:40.055716 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:33:40.058285 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:40.059641 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:40.061888 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:40.062590 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:40.062711 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:40.068579 (+ 5868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:33:40.068593 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:33:40.068609 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:33:40.068659 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:33:40.072888 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:40.073877 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:40.073889 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:40.073893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:40.074185 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:40.074712 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:40.074726 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:40.074730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:40.075119 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:33:40.075379 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:40.075699 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 14:33:40.075860 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:33:40.076953 25673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:33:40.175766 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 25470
I20260810 14:33:40.201983 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 25690
Aug 10 14:33:40 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:33:40.219571 26178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:40.202904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47532 (local address 127.6.46.194:33795)
0810 14:33:40.203123 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:40.203128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:40.203150 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:33:40.203307 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:40.203311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:40.203395 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:40.203491 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:33:40.204607 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:40.205322 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:40.206626 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:40.207038 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:40.207138 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:40.214432 (+ 7294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:33:40.214447 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:33:40.214462 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:33:40.214494 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:33:40.217207 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:40.217824 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:40.217831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:40.217834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:40.217953 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:40.218294 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:40.218297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:40.218299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:40.218664 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:33:40.218874 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:40.219093 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 14:33:40.219244 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 14:33:40.220149 25914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:33:40.306826 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 25690
I20260810 14:33:40.330330 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 25934
Aug 10 14:33:40 dist-test-slave-q9h9 krb5kdc[24511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:33:40.344920 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:33:40.331079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.6.46.195:35987)
0810 14:33:40.331265 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 14:33:40.331271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:33:40.331290 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:33:40.331462 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:33:40.331466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:33:40.331531 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:33:40.331630 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:33:40.332941 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:40.333631 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:40.335092 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:33:40.335479 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:33:40.335589 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:33:40.339793 (+ 4204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:33:40.339807 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:33:40.339824 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:33:40.339855 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:33:40.342341 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:40.342940 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:40.342947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:40.342949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:40.343073 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:33:40.343438 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:33:40.343441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:33:40.343443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:33:40.343813 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:33:40.344084 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 14:33:40.344395 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 14:33:40.344589 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:33:40.345569 26139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:33:40.430909 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 25934
I20260810 14:33:40.450742 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 25291
I20260810 14:33:40.451879 25387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:33:40.560186 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 25291
I20260810 14:33:40.587795 6331 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 14:33:40.933542 6331 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 14:33:40.933821 6331 mini_ranger.cc:66] Stopping Ranger...
I20260810 14:33:41.278131 6331 mini_ranger.cc:68] Stopped Ranger
2026-08-10 14:33:41.278 UTC [24566] LOG: received smart shutdown request
2026-08-10 14:33:41.281 UTC [24566] LOG: background worker "logical replication launcher" (PID 24581) exited with exit code 1
2026-08-10 14:33:41.281 UTC [24571] LOG: shutting down
2026-08-10 14:33:41.409 UTC [24566] LOG: database system is shut down
2026-08-10T14:33:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47779 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:33:41 dist-test-slave-q9h9 krb5kdc[26246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:33:41 dist-test-slave-q9h9 krb5kdc[26246](info): set up 2 sockets
Aug 10 14:33:41 dist-test-slave-q9h9 krb5kdc[26246](info): commencing operation
krb5kdc: starting...
W20260810 14:33:42.762704 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.200s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:33:42 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:33:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:33:42Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/rangeradmin_127.6.46.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/rangeradmin_127.6.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/rangerlookup_127.6.46.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/rangerlookup_127.6.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/HTTP_127.6.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/HTTP_127.6.46.212@KRBTEST.COM.keytab.
I20260810 14:33:43.061527 6331 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task50NkZQ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/postgres -l logfile start
2026-08-10 14:33:45.159 UTC [26295] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 14:33:45.159 UTC [26295] LOG: listening on IPv4 address "127.6.46.212", port 45511
2026-08-10 14:33:45.162 UTC [26295] LOG: listening on Unix socket "/tmp/.s.PGSQL.45511"
2026-08-10 14:33:45.182 UTC [26301] LOG: database system was shut down at 2026-08-10 14:33:44 UTC
2026-08-10 14:33:45.185 UTC [26295] LOG: database system is ready to accept connections
I20260810 14:33:46.283671 6331 mini_postgres.cc:96] Postgres bound to 45511
2026-08-10 14:33:46.305 UTC [26311] FATAL: database "slave" does not exist
127.6.46.212:45511 - accepting connections
I20260810 14:33:46.305619 6331 mini_ranger.cc:161] Starting Ranger...
I20260810 14:33:46.333606 6331 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 14:33:46.713481 6331 mini_ranger.cc:87] Created ranger Postgres database
I20260810 14:33:46.713582 6331 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-admin
2026-08-10 14:33:46,973 [I] DB FLAVOR :POSTGRES
2026-08-10 14:33:46,973 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:33:46,973 [I] Checking connection..
2026-08-10 14:33:46,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 14:33:47,227 [I] Checking connection passed.
2026-08-10 14:33:47,228 [I] --------- Verifying version history table ---------
2026-08-10 14:33:47,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:33:47,488 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:33:47,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:33:47,726 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 14:33:47,727 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 14:33:47,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 14:33:47.958 UTC [26420] WARNING: there is no transaction in progress
2026-08-10 14:33:47,965 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 14:33:47,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 14:33:48,215 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 14:33:48,215 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:33:48,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 14:33:48,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 14:33:48,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-q9h9.c.gcp-upstream.internal','N') ;"
2026-08-10 14:33:48,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:33:49,190 [I] Table x_portal_user does not exist in database ranger
2026-08-10 14:33:49,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:33:49,435 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 14:33:49,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:33:49,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:33:49,917 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 14:33:49,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 14:33:50.772 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.786 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.800 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.813 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.826 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.844 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.892 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.898 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.910 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.919 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.928 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.940 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.949 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.953 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.959 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:50.964 UTC [26594] WARNING: there is no transaction in progress
2026-08-10 14:33:51,483 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 14:33:51,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 14:33:51,722 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 14:33:51,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 14:33:51,973 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 14:33:51,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 14:33:52,231 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 14:33:52,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 14:33:52,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 14:33:52,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-q9h9.c.gcp-upstream.internal';"
2026-08-10 14:33:52,948 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 14:33:52,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 14:33:53,175 [I] DB_PATCHES have already been applied
I20260810 14:33:53.181427 6331 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-admin:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task50NkZQ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 14:33:53.181514 6331 mini_ranger.cc:193] Using host: 127.6.46.212
I20260810 14:33:53.200469 6331 mini_ranger.cc:227] Ranger admin URL: http://127.6.46.212:45759
Aug 10, 2026 2:33:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:33:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:33:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 2:33:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 2:33:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 2:33:54 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 2:33:54 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 14:33:54.441143 6331 mini_ranger.cc:160] Time spent starting Ranger: real 8.136s user 0.002s sys 0.049s
Aug 10, 2026 2:34:13 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 2:34:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 2:34:13 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 2:34:13 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 2:34:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 2:34:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 2:34:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 2:34:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 2:34:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 2:34:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 2:34:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 2:34:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 2:34:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 2:34:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 2:34:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 2:34:13 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 2:34:13 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 2:34:14 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 14:34:15.254496 6331 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.46.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.46.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/rangerkms_127.6.46.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.46.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/rangerkms_127.6.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.46.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.46.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.46.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/HTTP_127.6.46.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.46.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/HTTP_127.6.46.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 14:34:15 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372455, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 14:34:15.457883 6331 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 14:34:15.487725 6331 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 14:34:16.173417 6331 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 14:34:16.173521 6331 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-kms
I20260810 14:34:16.173563 6331 mini_ranger_kms.cc:227] Using postgres at 127.6.46.212:45511
2026-08-10 14:34:16,511 [I] DB FLAVOR :POSTGRES
2026-08-10 14:34:16,512 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 14:34:16,512 [I] Checking connection
2026-08-10 14:34:16,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 14:34:16,748 [I] connection success
2026-08-10 14:34:16,748 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 14:34:16,748 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 14:34:16,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 14:34:16,973 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 14:34:16,974 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 14:34:16,974 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 14:34:16,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.46.212:45511/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 14:34:17,218 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 14:34:17.650276 6331 mini_ranger_kms.cc:310] Created kms service
I20260810 14:34:17.812989 6331 mini_ranger_kms.cc:326] Created kudu user
I20260810 14:34:17.854140 6331 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 14:34:17 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372455, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:34:18.257455 6331 mini_ranger_kms.cc:384] Added ranger policy
I20260810 14:34:18.257787 6331 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-kms:/tmp/dist-test-task50NkZQ/build/asan/bin/postgresql.jar:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task50NkZQ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 14:34:18.257855 6331 mini_ranger_kms.cc:240] Using host: 127.6.46.212
I20260810 14:34:18.283210 6331 mini_ranger_kms.cc:276] Ranger KMS PID: 26873
I20260810 14:34:18.283326 6331 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.46.212:48045
Aug 10, 2026 2:34:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 2:34:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 2:34:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 2:34:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 2:34:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 2:34:19 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 14:34:20 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372460, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.46.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:20.961954 6331 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.504s user 0.066s sys 0.307s
I20260810 14:34:20.962245 6331 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 14:34:20.962436 6331 mini_ranger_kms.cc:401] 127.6.46.212:48045/kms/v1/keys
Aug 10 14:34:21 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372460, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.46.212@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
Aug 10 14:34:21 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372460, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.46.212@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
Aug 10 14:34:23 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:34:23.785368 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43951
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:43951
--ranger_config_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.46.212:48045/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:34:24.107631 26944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:24.107892 26944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:24.107975 26944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:24.114902 26944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:34:24.114995 26944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:24.115046 26944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:24.115106 26944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:34:24.115154 26944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:34:24.124634 26944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.46.212:48045/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:43951
--ranger_config_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:24.125718 26944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:24.127007 26944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:24.138319 26959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:24.138258 26955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:24.138530 26953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:24.139596 26944 server_base.cc:1034] running on GCE node
I20260810 14:34:24.140259 26944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:24.141480 26944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:24.142683 26944 hybrid_clock.cc:648] HybridClock initialized: now 1786372464142634 us; error 39 us; skew 500 ppm
I20260810 14:34:24.142999 26944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:24 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372464, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:24.149120 26944 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:34:24.150588 26944 webserver.cc:460] Webserver started at http://127.6.46.254:35079/ using document root <none> and password file <none>
I20260810 14:34:24.151181 26944 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:24.151280 26944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:24.151526 26944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:34:24 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372464, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:34:24.210861 26944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "c576a55b62784d878a86993e7a6ed601"
format_stamp: "Formatted at 2026-08-10 14:34:24 on dist-test-slave-q9h9"
server_key: "a9749003a679088a127a87600af59f59"
server_key_iv: "3ee33c344a49b6b56b0ecf0c0fafef31"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:24.211683 26944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "c576a55b62784d878a86993e7a6ed601"
format_stamp: "Formatted at 2026-08-10 14:34:24 on dist-test-slave-q9h9"
server_key: "a9749003a679088a127a87600af59f59"
server_key_iv: "3ee33c344a49b6b56b0ecf0c0fafef31"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:24.217216 26944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 14:34:24.232048 26969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:24.233644 26944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:34:24.233844 26944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "c576a55b62784d878a86993e7a6ed601"
format_stamp: "Formatted at 2026-08-10 14:34:24 on dist-test-slave-q9h9"
server_key: "a9749003a679088a127a87600af59f59"
server_key_iv: "3ee33c344a49b6b56b0ecf0c0fafef31"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:24.234069 26944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:24.249194 26944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:24.252856 26944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:24.253135 26944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:24.271095 26944 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:43951
I20260810 14:34:24.271139 27062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:43951 every 8 connection(s)
I20260810 14:34:24.272346 26944 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:34:24.275854 27064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:24.279243 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 26944
I20260810 14:34:24.279434 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:34:24.280082 6331 external_mini_cluster.cc:1428] Setting key 835eba298c5322a03850ad4a20dfb573
I20260810 14:34:24.286154 27064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601: Bootstrap starting.
I20260810 14:34:24.289470 27064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:24.290621 27064 log.cc:824] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:24.293606 27064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601: No bootstrap required, opened a new log
Aug 10 14:34:24 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:24.303124 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:24.281872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.6.46.254:43951)
0810 14:34:24.282599 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:24.282613 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:24.282664 (+ 51us) server_negotiation.cc:407] Connection header received
0810 14:34:24.284710 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:24.284733 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:24.285071 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:24.285468 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:24.287106 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:24.288372 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:24.290136 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:24.290798 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:24.290935 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:24.295324 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:24.295334 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:24.295356 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:24.295401 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:24.299035 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:24.299663 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:24.299671 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:24.299679 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:24.299865 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:24.300177 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:24.300182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:24.300185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:24.300868 (+ 683us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:24.301171 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:24.301455 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 14:34:24.301833 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":353,"threads_started":1}
I20260810 14:34:24.303678 27064 raft_consensus.cc:348] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c576a55b62784d878a86993e7a6ed601" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43951 } }
I20260810 14:34:24.304008 27064 raft_consensus.cc:374] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:24.304104 27064 raft_consensus.cc:729] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c576a55b62784d878a86993e7a6ed601, State: Initialized, Role: FOLLOWER
I20260810 14:34:24.304739 27064 consensus_queue.cc:260] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c576a55b62784d878a86993e7a6ed601" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43951 } }
I20260810 14:34:24.305013 27064 raft_consensus.cc:388] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:34:24.305163 27064 raft_consensus.cc:482] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:34:24.305333 27064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:24.307456 27064 raft_consensus.cc:504] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c576a55b62784d878a86993e7a6ed601" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43951 } }
I20260810 14:34:24.307866 27064 leader_election.cc:302] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c576a55b62784d878a86993e7a6ed601; no voters:
I20260810 14:34:24.308367 27064 leader_election.cc:288] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:34:24.308478 27072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:34:24.309207 27072 raft_consensus.cc:686] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 1 LEADER]: Becoming Leader. State: Replica: c576a55b62784d878a86993e7a6ed601, State: Running, Role: LEADER
I20260810 14:34:24.309724 27072 consensus_queue.cc:237] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c576a55b62784d878a86993e7a6ed601" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43951 } }
I20260810 14:34:24.310410 27064 sys_catalog.cc:564] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:34:24.312881 27073 sys_catalog.cc:455] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c576a55b62784d878a86993e7a6ed601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c576a55b62784d878a86993e7a6ed601" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43951 } } }
I20260810 14:34:24.312767 27075 sys_catalog.cc:455] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c576a55b62784d878a86993e7a6ed601. Latest consensus state: current_term: 1 leader_uuid: "c576a55b62784d878a86993e7a6ed601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c576a55b62784d878a86993e7a6ed601" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43951 } } }
I20260810 14:34:24.313750 27073 sys_catalog.cc:458] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:24.313920 27075 sys_catalog.cc:458] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:24.315383 27088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:34:24.320041 27088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:34:24.321934 27064 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 14:34:24.338577 27088 catalog_manager.cc:1347] Generated new cluster ID: 6785bc20b14945daad458cf5bbdca4c7
I20260810 14:34:24.338776 27088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:34:24.348224 27088 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:34:24.349714 27088 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:34:24.363322 27088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601: Generated new TSK 0
I20260810 14:34:24.364291 27088 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:34:25.089118 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.46.212:48045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43951
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:34:25 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372465, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 14:34:25.494086 27140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:25.494452 27140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:25.494671 27140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:25.505752 27140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:25.505924 27140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:25.506177 27140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:34:25.521502 27140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.46.212:48045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:25.523452 27140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:25.525264 27140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:25.556977 27156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:25.557866 27140 server_base.cc:1034] running on GCE node
W20260810 14:34:25.558317 27157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:25.564221 27161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:25.564996 27140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:25.566589 27140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:25.567929 27140 hybrid_clock.cc:648] HybridClock initialized: now 1786372465567805 us; error 100 us; skew 500 ppm
I20260810 14:34:25.568378 27140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:25 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372465, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:25.572549 27140 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:34:25.583488 27140 webserver.cc:460] Webserver started at http://127.6.46.193:40329/ using document root <none> and password file <none>
I20260810 14:34:25.584458 27140 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:25.584668 27140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:25.585078 27140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:34:25 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372465, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:34:25.622219 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "404628b4b522431e9472df2ccf391686"
format_stamp: "Formatted at 2026-08-10 14:34:25 on dist-test-slave-q9h9"
server_key: "a2f727f3846831fd0f1cdf00ed848777"
server_key_iv: "d08594918c4378cf872aa8acac15938f"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:25.628130 27140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "404628b4b522431e9472df2ccf391686"
format_stamp: "Formatted at 2026-08-10 14:34:25 on dist-test-slave-q9h9"
server_key: "a2f727f3846831fd0f1cdf00ed848777"
server_key_iv: "d08594918c4378cf872aa8acac15938f"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:25.643038 27140 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.005s sys 0.000s
I20260810 14:34:25.670056 27169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:25.688336 27140 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.002s sys 0.000s
I20260810 14:34:25.688624 27140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "404628b4b522431e9472df2ccf391686"
format_stamp: "Formatted at 2026-08-10 14:34:25 on dist-test-slave-q9h9"
server_key: "a2f727f3846831fd0f1cdf00ed848777"
server_key_iv: "d08594918c4378cf872aa8acac15938f"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:25.688998 27140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:25.759788 27140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:25.777150 27140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:25.777479 27140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:25.778738 27140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:25.780611 27140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:25.780758 27140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 14:34:25.780957 27140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:25.781105 27140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:34:25 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372465, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:34:25.908483 27140 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:35759
I20260810 14:34:25.908535 27347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:35759 every 8 connection(s)
I20260810 14:34:25.910157 27140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:34:25.915885 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 27140
I20260810 14:34:25.916108 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:34:25.916924 6331 external_mini_cluster.cc:1428] Setting key 88dd0dd9ae421bd72536f52ac7aead5d
Aug 10 14:34:25 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372465, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
I20260810 14:34:25.975193 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:25.913271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:48695 (local address 127.6.46.254:43951)
0810 14:34:25.944263 (+ 30992us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:25.944271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:25.951267 (+ 6996us) server_negotiation.cc:407] Connection header received
0810 14:34:25.953621 (+ 2354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:25.953628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:25.953737 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:25.953820 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:25.956102 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:25.957125 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:25.959494 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:25.960105 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:25.960201 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:25.964716 (+ 4515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:25.964723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:25.964728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:25.964783 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:25.968149 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:25.968833 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:25.968839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:25.968843 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:25.968992 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:25.969419 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:25.969424 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:25.969427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:25.969782 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:25.969930 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:25.974560 (+ 4630us) server_negotiation.cc:299] Negotiation successful
0810 14:34:25.974819 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30825,"thread_start_us":30995,"threads_started":1}
I20260810 14:34:25.982163 27358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:25.913808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:43951 (local address 127.6.46.193:48695)
0810 14:34:25.950694 (+ 36886us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:25.950751 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:25.953242 (+ 2491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:25.954407 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:25.954420 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:25.955080 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:25.955882 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:25.955907 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:25.957333 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:25.957337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:25.959335 (+ 1998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:25.959351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:25.960585 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:25.960593 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:25.960893 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:25.961676 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:25.961711 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:25.964438 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:25.968283 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:25.968293 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:25.968309 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:25.968705 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:25.969099 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:25.969104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:25.969108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:25.969300 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:25.973631 (+ 4331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:25.973638 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:25.974128 (+ 490us) client_negotiation.cc:769] Sending connection context
0810 14:34:25.975727 (+ 1599us) client_negotiation.cc:240] Negotiation successful
0810 14:34:25.976241 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36538,"thread_start_us":36305,"threads_started":1}
I20260810 14:34:25.984050 27353 heartbeater.cc:347] Connected to a master server at 127.6.46.254:43951
I20260810 14:34:25.984386 27353 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:25.985275 27353 heartbeater.cc:511] Master 127.6.46.254:43951 requested a full tablet report, sending...
I20260810 14:34:25.987836 27009 ts_manager.cc:194] Registered new tserver with Master: 404628b4b522431e9472df2ccf391686 (127.6.46.193:35759)
I20260810 14:34:25.989897 27009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:48695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:34:26.005405 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.46.212:48045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43951
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:34:26 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372465, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
W20260810 14:34:26.464504 27362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:26.464797 27362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:26.464913 27362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:26.475471 27362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:26.475667 27362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:26.476153 27362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:34:26.488725 27362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.46.212:48045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:26.489956 27362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:26.491377 27362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:26.505908 27385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:26.506352 27386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:26.506579 27388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:26.507227 27362 server_base.cc:1034] running on GCE node
I20260810 14:34:26.507741 27362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:26.508563 27362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:26.509763 27362 hybrid_clock.cc:648] HybridClock initialized: now 1786372466509714 us; error 40 us; skew 500 ppm
I20260810 14:34:26.510062 27362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:26 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372466, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:26.513075 27362 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:34:26.514411 27362 webserver.cc:460] Webserver started at http://127.6.46.194:34975/ using document root <none> and password file <none>
I20260810 14:34:26.515048 27362 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:26.515167 27362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:26.515488 27362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:34:26 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372466, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:34:26.529049 27362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "c06ed25bbe574ee98a8b2f316f8e9af8"
format_stamp: "Formatted at 2026-08-10 14:34:26 on dist-test-slave-q9h9"
server_key: "ce2fc77a5cd570d35afde8309c0dd5f4"
server_key_iv: "e0e5ebfbb9d9af77f48ac3cf3e2ab35e"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:26.529847 27362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "c06ed25bbe574ee98a8b2f316f8e9af8"
format_stamp: "Formatted at 2026-08-10 14:34:26 on dist-test-slave-q9h9"
server_key: "ce2fc77a5cd570d35afde8309c0dd5f4"
server_key_iv: "e0e5ebfbb9d9af77f48ac3cf3e2ab35e"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:26.535126 27362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 14:34:26.548884 27397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:26.550179 27362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:34:26.550328 27362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "c06ed25bbe574ee98a8b2f316f8e9af8"
format_stamp: "Formatted at 2026-08-10 14:34:26 on dist-test-slave-q9h9"
server_key: "ce2fc77a5cd570d35afde8309c0dd5f4"
server_key_iv: "e0e5ebfbb9d9af77f48ac3cf3e2ab35e"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:26.550518 27362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:26.577822 27362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:26.581404 27362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:26.581686 27362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:26.582628 27362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:26.583925 27362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:26.584002 27362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:26.584113 27362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:26.584167 27362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:26.613220 27362 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:37575
I20260810 14:34:26.613255 27590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:37575 every 8 connection(s)
I20260810 14:34:26.614439 27362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:34:26.616920 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 27362
I20260810 14:34:26.617113 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:34:26.617645 6331 external_mini_cluster.cc:1428] Setting key e405ed5076ff5af970d7c21ab627ffde
Aug 10 14:34:26 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372466, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:26.634191 27593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:26.616816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:58643 (local address 127.6.46.254:43951)
0810 14:34:26.617259 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:26.617264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:26.618513 (+ 1249us) server_negotiation.cc:407] Connection header received
0810 14:34:26.620554 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:26.620559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:26.620666 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:26.620760 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:26.622256 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:26.622996 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:26.624503 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:26.624957 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:26.625069 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:26.628414 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:26.628419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:26.628422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:26.628453 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:26.631263 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:26.631938 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:26.631942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:26.631944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:26.632068 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:26.632500 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:26.632504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:26.632506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:26.632767 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:26.632900 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:26.633551 (+ 651us) server_negotiation.cc:299] Negotiation successful
0810 14:34:26.633779 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":242,"threads_started":1}
I20260810 14:34:26.635263 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:26.617472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:43951 (local address 127.6.46.194:58643)
0810 14:34:26.618348 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:26.618396 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:26.620291 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:26.621049 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:26.621061 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:26.621498 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:26.622089 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:26.622112 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:26.623159 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:26.623165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:26.624367 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:26.624379 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:26.625181 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:26.625191 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:26.625436 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:26.626106 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:26.626130 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:26.628211 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:26.631396 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:26.631402 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:26.631413 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:26.631809 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:26.632175 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:26.632178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:26.632181 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:26.632369 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:26.632892 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:26.632899 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:26.633189 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 14:34:26.633412 (+ 223us) client_negotiation.cc:240] Negotiation successful
0810 14:34:26.633883 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":352,"threads_started":1}
I20260810 14:34:26.636718 27591 heartbeater.cc:347] Connected to a master server at 127.6.46.254:43951
I20260810 14:34:26.637023 27591 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:26.637827 27591 heartbeater.cc:511] Master 127.6.46.254:43951 requested a full tablet report, sending...
I20260810 14:34:26.639426 27009 ts_manager.cc:194] Registered new tserver with Master: c06ed25bbe574ee98a8b2f316f8e9af8 (127.6.46.194:37575)
I20260810 14:34:26.640479 27009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:58643
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:34:26.674269 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.46.212:48045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43951
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:34:26.988550 27598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:26.988821 27598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:26.988952 27598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 14:34:26.993557 27353 heartbeater.cc:503] Master 127.6.46.254:43951 was elected leader, sending a full tablet report...
W20260810 14:34:26.996423 27598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:26.996511 27598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:26.996619 27598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:34:27.005622 27598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.46.212:48045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:27.006699 27598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:27.007804 27598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:27.022101 27607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:27.022085 27609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:27.022667 27611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:27.023625 27598 server_base.cc:1034] running on GCE node
I20260810 14:34:27.024113 27598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:27.024914 27598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:27.026100 27598 hybrid_clock.cc:648] HybridClock initialized: now 1786372467026061 us; error 26 us; skew 500 ppm
I20260810 14:34:27.026369 27598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:27 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372467, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:27.029207 27598 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:34:27.030467 27598 webserver.cc:460] Webserver started at http://127.6.46.195:39121/ using document root <none> and password file <none>
I20260810 14:34:27.031054 27598 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:27.031147 27598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:27.031378 27598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 14:34:27 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372467, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:34:27.047160 27598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "513efccc57454ea89806c41e5d94bfaa"
format_stamp: "Formatted at 2026-08-10 14:34:27 on dist-test-slave-q9h9"
server_key: "1f0805fedae8bfac7bfc3cfe3b884773"
server_key_iv: "612439ed5cab74ecc69d40772e15322c"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:27.048175 27598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "513efccc57454ea89806c41e5d94bfaa"
format_stamp: "Formatted at 2026-08-10 14:34:27 on dist-test-slave-q9h9"
server_key: "1f0805fedae8bfac7bfc3cfe3b884773"
server_key_iv: "612439ed5cab74ecc69d40772e15322c"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:27.054677 27598 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260810 14:34:27.071274 27619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:27.072937 27598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:34:27.073141 27598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "513efccc57454ea89806c41e5d94bfaa"
format_stamp: "Formatted at 2026-08-10 14:34:27 on dist-test-slave-q9h9"
server_key: "1f0805fedae8bfac7bfc3cfe3b884773"
server_key_iv: "612439ed5cab74ecc69d40772e15322c"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:27.073370 27598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:27.098588 27598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:27.102831 27598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:27.103080 27598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:27.103883 27598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:27.105137 27598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:27.105206 27598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:27.105296 27598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:27.105340 27598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:27.134691 27598 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:35153
I20260810 14:34:27.134764 27807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:35153 every 8 connection(s)
I20260810 14:34:27.136063 27598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:34:27.144369 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 27598
I20260810 14:34:27.144521 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:34:27.144975 6331 external_mini_cluster.cc:1428] Setting key 35222fd4f0c2958651d616d411a26d59
Aug 10 14:34:27 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372467, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:27.154438 27810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:27.138544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:41325 (local address 127.6.46.254:43951)
0810 14:34:27.139017 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:27.139022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:27.139991 (+ 969us) server_negotiation.cc:407] Connection header received
0810 14:34:27.141721 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:27.141727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:27.141831 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:27.141926 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:27.143291 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:27.143993 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:27.145623 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:27.146222 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:27.146356 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:27.149542 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:27.149547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:27.149550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:27.149580 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:27.151919 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:27.152570 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:27.152574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:27.152576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:27.152705 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:27.153057 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:27.153060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:27.153062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:27.153318 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:27.153429 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:27.154000 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 14:34:27.154198 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":252,"threads_started":1}
I20260810 14:34:27.155272 27813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:27.139003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:43951 (local address 127.6.46.195:41325)
0810 14:34:27.139829 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:27.139874 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:27.141462 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:27.142192 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:27.142200 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:27.142594 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:27.143131 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:27.143147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:27.144150 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:27.144154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:27.145450 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:27.145477 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:27.146421 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:27.146426 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:27.146627 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:27.147252 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:27.147273 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:27.149326 (+ 2053us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:27.152035 (+ 2709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:27.152042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:27.152052 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:27.152443 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:27.152792 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:27.152795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:27.152797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:27.152975 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:27.153408 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:27.153413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:27.153705 (+ 292us) client_negotiation.cc:769] Sending connection context
0810 14:34:27.153908 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 14:34:27.154286 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":344,"threads_started":1}
I20260810 14:34:27.156497 27808 heartbeater.cc:347] Connected to a master server at 127.6.46.254:43951
I20260810 14:34:27.156734 27808 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:27.157348 27808 heartbeater.cc:511] Master 127.6.46.254:43951 requested a full tablet report, sending...
I20260810 14:34:27.158663 27009 ts_manager.cc:194] Registered new tserver with Master: 513efccc57454ea89806c41e5d94bfaa (127.6.46.195:35153)
I20260810 14:34:27.159585 27009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:41325
I20260810 14:34:27.160112 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:34:27.161717 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 27140
Aug 10 14:34:27 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:34:27.176406 27815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:27.162454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44142 (local address 127.6.46.193:35759)
0810 14:34:27.162961 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:27.162967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:27.162988 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:34:27.163042 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:27.163047 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:27.163200 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:27.163311 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:27.164658 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:27.165645 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:27.167044 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:27.167553 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:27.167645 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:27.171444 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:27.171455 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:27.171466 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:27.171509 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:27.173926 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:27.174679 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:27.174685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:27.174687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:27.174838 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:27.175184 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:27.175187 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:27.175189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:27.175485 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:27.175705 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:27.175941 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 14:34:27.176141 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":233,"threads_started":1}
I20260810 14:34:27.177554 27338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:27.257746 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 27140
I20260810 14:34:27.277763 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 27362
Aug 10 14:34:27 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:34:27.293642 27820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:27.278724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39346 (local address 127.6.46.194:37575)
0810 14:34:27.279288 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:27.279295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:27.279321 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:34:27.279395 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:27.279401 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:27.279610 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:27.279793 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:27.281110 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:27.282278 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:27.283643 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:27.284187 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:27.284299 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:27.288572 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:27.288584 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:27.288598 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:27.288641 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:27.291243 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:27.291860 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:27.291867 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:27.291869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:27.292002 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:27.292316 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:27.292320 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:27.292322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:27.292658 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:27.292880 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:27.293109 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 14:34:27.293303 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":288,"threads_started":1}
I20260810 14:34:27.295032 27586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:27.376557 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 27362
I20260810 14:34:27.398797 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 27598
Aug 10 14:34:27 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:34:27.414186 27823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:27.399852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47838 (local address 127.6.46.195:35153)
0810 14:34:27.400499 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:27.400506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:27.400536 (+ 30us) server_negotiation.cc:407] Connection header received
0810 14:34:27.400600 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:27.400606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:27.400824 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:27.400952 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:27.402253 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:27.403260 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:27.404422 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:27.404957 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:27.405056 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:27.409138 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:27.409157 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:27.409170 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:27.409203 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:27.411798 (+ 2595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:27.412405 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:27.412411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:27.412413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:27.412534 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:27.412827 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:27.412829 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:27.412831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:27.413161 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:27.413398 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:27.413546 (+ 148us) server_negotiation.cc:299] Negotiation successful
0810 14:34:27.413736 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":336,"threads_started":1}
I20260810 14:34:27.415618 27799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:27.495144 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 27598
I20260810 14:34:27.516435 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 26944
I20260810 14:34:27.517669 27057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:27.620814 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 26944
I20260810 14:34:27.660497 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43951
--webserver_interface=127.6.46.254
--webserver_port=35079
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:43951
--ranger_config_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.46.212:48045/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.46.212:48045/kms with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:34:27.977437 27829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:27.977706 27829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:27.977829 27829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:27.984902 27829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:34:27.985018 27829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:27.985111 27829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:27.985196 27829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:34:27.985262 27829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:34:27.994671 27829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.46.212:48045/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:43951
--ranger_config_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=35079
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:27.995764 27829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:27.996953 27829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:28.007899 27838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:28.007846 27840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:28.008306 27837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:28.008807 27829 server_base.cc:1034] running on GCE node
I20260810 14:34:28.009469 27829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:28.010727 27829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:28.011927 27829 hybrid_clock.cc:648] HybridClock initialized: now 1786372468011882 us; error 40 us; skew 500 ppm
I20260810 14:34:28.012233 27829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:28 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372468, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:28.015106 27829 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:34:28.016436 27829 webserver.cc:460] Webserver started at http://127.6.46.254:35079/ using document root <none> and password file <none>
I20260810 14:34:28.017007 27829 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:28.017138 27829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 14:34:28 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372468, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:34:28.066594 27829 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 14:34:28.069355 27854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:28.070792 27829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:34:28.070971 27829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "c576a55b62784d878a86993e7a6ed601"
format_stamp: "Formatted at 2026-08-10 14:34:24 on dist-test-slave-q9h9"
server_key: "a9749003a679088a127a87600af59f59"
server_key_iv: "3ee33c344a49b6b56b0ecf0c0fafef31"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:28.071869 27829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:28.086656 27829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:28.089794 27829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:28.090075 27829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:28.106628 27829 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:43951
I20260810 14:34:28.106647 27946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:43951 every 8 connection(s)
I20260810 14:34:28.107929 27829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:34:28.113467 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 27829
I20260810 14:34:28.114296 27948 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:34:28.114319 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.46.212:48045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:35759
--local_ip_for_outbound_sockets=127.6.46.193
--tserver_master_addrs=127.6.46.254:43951
--webserver_port=40329
--webserver_interface=127.6.46.193
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.46.212:48045/kms with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:34:28.117408 27948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601: Bootstrap starting.
I20260810 14:34:28.135068 27948 log.cc:824] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:28.143247 27948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:34:28.143702 27948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601: Bootstrap complete.
I20260810 14:34:28.152017 27948 raft_consensus.cc:348] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c576a55b62784d878a86993e7a6ed601" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43951 } }
I20260810 14:34:28.152820 27948 raft_consensus.cc:729] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c576a55b62784d878a86993e7a6ed601, State: Initialized, Role: FOLLOWER
I20260810 14:34:28.153332 27948 consensus_queue.cc:260] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c576a55b62784d878a86993e7a6ed601" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43951 } }
I20260810 14:34:28.153534 27948 raft_consensus.cc:388] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:34:28.153642 27948 raft_consensus.cc:482] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:34:28.153796 27948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:34:28.157315 27948 raft_consensus.cc:504] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c576a55b62784d878a86993e7a6ed601" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43951 } }
I20260810 14:34:28.157755 27948 leader_election.cc:302] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c576a55b62784d878a86993e7a6ed601; no voters:
I20260810 14:34:28.158341 27948 leader_election.cc:288] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 14:34:28.158468 27955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:34:28.159514 27955 raft_consensus.cc:686] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [term 2 LEADER]: Becoming Leader. State: Replica: c576a55b62784d878a86993e7a6ed601, State: Running, Role: LEADER
I20260810 14:34:28.160101 27955 consensus_queue.cc:237] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c576a55b62784d878a86993e7a6ed601" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43951 } }
I20260810 14:34:28.160342 27948 sys_catalog.cc:564] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:34:28.167371 27957 sys_catalog.cc:455] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c576a55b62784d878a86993e7a6ed601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c576a55b62784d878a86993e7a6ed601" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43951 } } }
I20260810 14:34:28.167769 27957 sys_catalog.cc:458] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:28.168292 27963 sys_catalog.cc:455] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c576a55b62784d878a86993e7a6ed601. Latest consensus state: current_term: 2 leader_uuid: "c576a55b62784d878a86993e7a6ed601" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c576a55b62784d878a86993e7a6ed601" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43951 } } }
I20260810 14:34:28.168556 27963 sys_catalog.cc:458] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601 [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:28.172978 27974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:34:28.179422 27974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:34:28.180845 27974 catalog_manager.cc:1259] Loaded cluster ID: 6785bc20b14945daad458cf5bbdca4c7
I20260810 14:34:28.180980 27974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:34:28.185114 27974 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:34:28.186961 27974 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601: Loaded TSK: 0
I20260810 14:34:28.188547 27974 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:34:28.190980 27948 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260810 14:34:28.534194 27953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:28.534648 27953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:28.534866 27953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:28.545717 27953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:28.545919 27953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:28.546175 27953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:34:28.558561 27953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.46.212:48045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:35759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=40329
--tserver_master_addrs=127.6.46.254:43951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:28.560261 27953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:28.561683 27953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:28.587800 28017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:28.590160 28020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:28.590814 28018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:28.591602 27953 server_base.cc:1034] running on GCE node
I20260810 14:34:28.592546 27953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:28.602226 27953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:28.603513 27953 hybrid_clock.cc:648] HybridClock initialized: now 1786372468603427 us; error 81 us; skew 500 ppm
I20260810 14:34:28.603821 27953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:28 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372468, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:28.606974 27953 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:34:28.612419 27953 webserver.cc:460] Webserver started at http://127.6.46.193:40329/ using document root <none> and password file <none>
I20260810 14:34:28.613111 27953 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:28.613250 27953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 14:34:28 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372468, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:34:28.649009 27953 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.003s sys 0.000s
I20260810 14:34:28.655174 28037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:28.656641 27953 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.003s
I20260810 14:34:28.656798 27953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "404628b4b522431e9472df2ccf391686"
format_stamp: "Formatted at 2026-08-10 14:34:25 on dist-test-slave-q9h9"
server_key: "a2f727f3846831fd0f1cdf00ed848777"
server_key_iv: "d08594918c4378cf872aa8acac15938f"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:28.658140 27953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:28.693015 27953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:28.698190 27953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:28.698573 27953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:28.702539 27953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:28.704304 27953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:28.704473 27953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260810 14:34:28.704645 27953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:28.704785 27953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:28.771523 27953 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:35759
I20260810 14:34:28.771569 28226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:35759 every 8 connection(s)
I20260810 14:34:28.773101 27953 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:34:28.780174 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 27953
I20260810 14:34:28.780973 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.46.212:48045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:37575
--local_ip_for_outbound_sockets=127.6.46.194
--tserver_master_addrs=127.6.46.254:43951
--webserver_port=34975
--webserver_interface=127.6.46.194
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.46.212:48045/kms with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:34:28 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372468, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:28.803572 28235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:28.776785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:43951 (local address 127.6.46.193:37631)
0810 14:34:28.778206 (+ 1421us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:28.778253 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:28.780568 (+ 2315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:28.786226 (+ 5658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:28.786236 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:28.786822 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:28.787537 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:28.787563 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:28.789661 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:28.789666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:28.791400 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:28.791415 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:28.792675 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:28.792682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:28.792997 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:28.793698 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:28.793728 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:28.796029 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:28.799374 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:28.799382 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:28.799396 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:28.799907 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:28.800353 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:28.800358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:28.800363 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:28.800595 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:28.801434 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:28.801440 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:28.801729 (+ 289us) client_negotiation.cc:769] Sending connection context
0810 14:34:28.802058 (+ 329us) client_negotiation.cc:240] Negotiation successful
0810 14:34:28.802584 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"thread_start_us":937,"threads_started":1}
I20260810 14:34:28.805394 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:28.776637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:37631 (local address 127.6.46.254:43951)
0810 14:34:28.784207 (+ 7570us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:28.784224 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:28.784271 (+ 47us) server_negotiation.cc:407] Connection header received
0810 14:34:28.784668 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:28.784687 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:28.785143 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:28.785522 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:28.787772 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:28.789465 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:28.791560 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:28.792445 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:28.792613 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:28.796284 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:28.796291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:28.796304 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:28.796337 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:28.799245 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:28.800069 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:28.800076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:28.800081 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:28.800240 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:28.800737 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:28.800743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:28.800746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:28.801308 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:28.801483 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:28.804277 (+ 2794us) server_negotiation.cc:299] Negotiation successful
0810 14:34:28.804545 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7331,"thread_start_us":7087,"threads_started":1}
I20260810 14:34:28.808274 28229 heartbeater.cc:347] Connected to a master server at 127.6.46.254:43951
I20260810 14:34:28.808574 28229 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:28.809233 28229 heartbeater.cc:511] Master 127.6.46.254:43951 requested a full tablet report, sending...
W20260810 14:34:28.811852 28229 heartbeater.cc:644] Failed to heartbeat to 127.6.46.254:43951 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 14:34:28.812062 28229 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:28.812300 28229 heartbeater.cc:511] Master 127.6.46.254:43951 requested a full tablet report, sending...
I20260810 14:34:28.813374 28229 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:28.813609 28229 heartbeater.cc:511] Master 127.6.46.254:43951 requested a full tablet report, sending...
W20260810 14:34:28.814554 28229 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:34:29.203612 28237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:29.203951 28237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:29.204154 28237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:29.213678 28237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:29.213812 28237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:29.214003 28237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:34:29.227623 28237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.46.212:48045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:37575
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=34975
--tserver_master_addrs=127.6.46.254:43951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:29.229180 28237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:29.230829 28237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:29.246616 28265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:29.246646 28267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:29.246582 28270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:29.248342 28237 server_base.cc:1034] running on GCE node
I20260810 14:34:29.248957 28237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:29.249959 28237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:29.251178 28237 hybrid_clock.cc:648] HybridClock initialized: now 1786372469251113 us; error 57 us; skew 500 ppm
I20260810 14:34:29.251480 28237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:29 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372469, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:29.258131 28237 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:34:29.259579 28237 webserver.cc:460] Webserver started at http://127.6.46.194:34975/ using document root <none> and password file <none>
I20260810 14:34:29.260247 28237 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:29.260380 28237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 14:34:29 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372469, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:34:29.305614 28237 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.009s sys 0.000s
I20260810 14:34:29.309746 28286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:29.311446 28237 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260810 14:34:29.311676 28237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "c06ed25bbe574ee98a8b2f316f8e9af8"
format_stamp: "Formatted at 2026-08-10 14:34:26 on dist-test-slave-q9h9"
server_key: "ce2fc77a5cd570d35afde8309c0dd5f4"
server_key_iv: "e0e5ebfbb9d9af77f48ac3cf3e2ab35e"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:29.313103 28237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:29.368422 28237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:29.373474 28237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:29.373799 28237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:29.375408 28237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:29.377108 28237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:29.377281 28237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 14:34:29.378202 28237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:29.378333 28237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 14:34:29 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786372469, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:29.455370 28237 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:37575
I20260810 14:34:29.455411 28469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:37575 every 8 connection(s)
I20260810 14:34:29.457208 28237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:34:29.458097 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 28237
I20260810 14:34:29.458915 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.46.212:48045/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:35153
--local_ip_for_outbound_sockets=127.6.46.195
--tserver_master_addrs=127.6.46.254:43951
--webserver_port=39121
--webserver_interface=127.6.46.195
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.46.212:48045/kms with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:34:29 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372469, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:29.495424 28476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:29.461925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:53997 (local address 127.6.46.254:43951)
0810 14:34:29.470426 (+ 8501us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:29.470434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:29.474711 (+ 4277us) server_negotiation.cc:407] Connection header received
0810 14:34:29.477192 (+ 2481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:29.477198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:29.477289 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:29.477380 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:29.479641 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:29.480795 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:29.483194 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:29.483809 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:29.483910 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:29.488216 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:29.488223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:29.488228 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:29.488268 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:29.491728 (+ 3460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:29.492619 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:29.492624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:29.492627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:29.492778 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:29.493340 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:29.493345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:29.493349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:29.493678 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:29.493820 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:29.494823 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0810 14:34:29.495041 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8351,"thread_start_us":8234,"threads_started":1}
I20260810 14:34:29.497839 28474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:29.461324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:43951 (local address 127.6.46.194:53997)
0810 14:34:29.474544 (+ 13220us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:29.474582 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:29.476853 (+ 2271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:29.477934 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:29.477945 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:29.478608 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:29.479439 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:29.479464 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:29.481002 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:29.481007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:29.483021 (+ 2014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:29.483037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:29.484243 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:29.484250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:29.484515 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:29.485297 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:29.485327 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:29.487963 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:29.491884 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:29.491892 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:29.491909 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:29.492481 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:29.492926 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:29.492931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:29.492934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:29.493209 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:29.493916 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:29.493923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:29.494378 (+ 455us) client_negotiation.cc:769] Sending connection context
0810 14:34:29.495961 (+ 1583us) client_negotiation.cc:240] Negotiation successful
0810 14:34:29.496508 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12954,"thread_start_us":12765,"threads_started":1}
I20260810 14:34:29.499785 28471 heartbeater.cc:347] Connected to a master server at 127.6.46.254:43951
I20260810 14:34:29.500106 28471 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:29.500964 28471 heartbeater.cc:511] Master 127.6.46.254:43951 requested a full tablet report, sending...
I20260810 14:34:29.503012 27885 ts_manager.cc:194] Registered new tserver with Master: c06ed25bbe574ee98a8b2f316f8e9af8 (127.6.46.194:37575)
I20260810 14:34:29.505129 27885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:53997
I20260810 14:34:29.814705 28229 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:29.815173 28229 heartbeater.cc:511] Master 127.6.46.254:43951 requested a full tablet report, sending...
I20260810 14:34:29.816565 27885 ts_manager.cc:194] Registered new tserver with Master: 404628b4b522431e9472df2ccf391686 (127.6.46.193:35759)
I20260810 14:34:29.817723 27885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:37631
Aug 10 14:34:29 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372469, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
W20260810 14:34:29.905041 28478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:29.905421 28478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:29.905587 28478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:29.916750 28478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:29.916875 28478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:29.917058 28478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:34:29.927160 28478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.46.212:48045/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:35153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=39121
--tserver_master_addrs=127.6.46.254:43951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:29.928412 28478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:29.929677 28478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:29.943567 28488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:29.943567 28486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:29.944448 28491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:29.944589 28478 server_base.cc:1034] running on GCE node
I20260810 14:34:29.945163 28478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:29.946084 28478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:29.947304 28478 hybrid_clock.cc:648] HybridClock initialized: now 1786372469947244 us; error 51 us; skew 500 ppm
I20260810 14:34:29.947583 28478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:29 dist-test-slave-q9h9 krb5kdc[26246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372469, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:29.950997 28478 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:34:29.958338 28478 webserver.cc:460] Webserver started at http://127.6.46.195:39121/ using document root <none> and password file <none>
I20260810 14:34:29.959014 28478 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:29.959167 28478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 14:34:29 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372469, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:34:29.982628 28478 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20260810 14:34:29.985325 28504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:29.986726 28478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:34:29.986878 28478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "513efccc57454ea89806c41e5d94bfaa"
format_stamp: "Formatted at 2026-08-10 14:34:27 on dist-test-slave-q9h9"
server_key: "1f0805fedae8bfac7bfc3cfe3b884773"
server_key_iv: "612439ed5cab74ecc69d40772e15322c"
server_key_version: "kuduclusterkey@0"
I20260810 14:34:29.987830 28478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:30.001839 28478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:30.005502 28478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:30.005760 28478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:30.006769 28478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:30.008219 28478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:30.008328 28478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:30.008431 28478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:30.008505 28478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:30.074398 28478 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:35153
I20260810 14:34:30.074579 28690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:35153 every 8 connection(s)
I20260810 14:34:30.076547 28478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:34:30.084214 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 28478
Aug 10 14:34:30 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372469, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:30.149904 28698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.083602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:45821 (local address 127.6.46.254:43951)
0810 14:34:30.125766 (+ 42164us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:30.125774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:30.128554 (+ 2780us) server_negotiation.cc:407] Connection header received
0810 14:34:30.131142 (+ 2588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:30.131146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:30.131244 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:30.131323 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:30.133413 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.134544 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.137505 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.138203 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.138299 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.142643 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:30.142651 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:30.142656 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:30.142694 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:30.146181 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:30.147107 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:30.147113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:30.147117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:30.147296 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:30.147844 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:30.147849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:30.147852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:30.148187 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:30.148312 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:30.149244 (+ 932us) server_negotiation.cc:299] Negotiation successful
0810 14:34:30.149480 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":974,"server-negotiator.queue_time_us":42032,"spinlock_wait_cycles":25984,"thread_start_us":41915,"threads_started":1}
I20260810 14:34:30.152091 28697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.079915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:43951 (local address 127.6.46.195:45821)
0810 14:34:30.128039 (+ 48124us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:30.128100 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:30.130795 (+ 2695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:30.131800 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:30.131812 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:30.132395 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:30.133202 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:30.133228 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.134754 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.134759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:30.137352 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:30.137367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.138650 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.138658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:30.138949 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.139624 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:30.139655 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:30.142344 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:30.146348 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:30.146358 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:30.146374 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:30.146946 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:30.147436 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:30.147441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:30.147445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:30.147707 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:30.148397 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:30.148405 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:30.148814 (+ 409us) client_negotiation.cc:769] Sending connection context
0810 14:34:30.150218 (+ 1404us) client_negotiation.cc:240] Negotiation successful
0810 14:34:30.150745 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47804,"thread_start_us":47580,"threads_started":1}
Aug 10 14:34:30 dist-test-slave-q9h9 krb5kdc[26246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372469, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for HTTP/127.6.46.212@KRBTEST.COM
I20260810 14:34:30.153793 28691 heartbeater.cc:347] Connected to a master server at 127.6.46.254:43951
I20260810 14:34:30.154162 28691 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:30.155107 28691 heartbeater.cc:511] Master 127.6.46.254:43951 requested a full tablet report, sending...
I20260810 14:34:30.156912 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.085575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52046 (local address 127.6.46.254:43951)
0810 14:34:30.126142 (+ 40567us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:30.126148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:30.126948 (+ 800us) server_negotiation.cc:407] Connection header received
0810 14:34:30.127029 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:30.127035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:30.127166 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:30.127269 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:30.131112 (+ 3843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.132194 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.134692 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.135315 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.135438 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.146983 (+ 11545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:30.146991 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:30.146995 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:30.147169 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:30.151185 (+ 4016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:30.152074 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:30.152080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:30.152082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:30.152245 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:30.154562 (+ 2317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:30.154567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:30.154570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:30.154922 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:30.155188 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:30.156366 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0810 14:34:30.156581 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40459,"thread_start_us":40322,"threads_started":1}
I20260810 14:34:30.156984 27885 ts_manager.cc:194] Registered new tserver with Master: 513efccc57454ea89806c41e5d94bfaa (127.6.46.195:35153)
I20260810 14:34:30.164436 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:34:30.166230 27885 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:45821
I20260810 14:34:30.221174 28701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.195475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52062 (local address 127.6.46.254:43951)
0810 14:34:30.195650 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:30.195657 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:30.202615 (+ 6958us) server_negotiation.cc:407] Connection header received
0810 14:34:30.202815 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:30.202821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:30.202921 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:30.203031 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:30.205007 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.205745 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.207850 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.208466 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.208556 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.212256 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:30.212263 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:30.212268 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:30.212308 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:30.215841 (+ 3533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:30.218593 (+ 2752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:30.218599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:30.218604 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:30.218759 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:30.219320 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:30.219325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:30.219329 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:30.219693 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:30.219924 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:30.220578 (+ 654us) server_negotiation.cc:299] Negotiation successful
0810 14:34:30.220811 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:34:30.227795 27885 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:34:30.238950 27885 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:34:30.299487 28721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.270825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:35153 (local address 127.0.0.1:33838)
0810 14:34:30.287900 (+ 17075us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:30.287948 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:30.288845 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:30.289970 (+ 1125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:30.289977 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:30.290007 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:30.290396 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:30.290409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.292562 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.292568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:30.295065 (+ 2497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:30.295081 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.298543 (+ 3462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.298550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:30.298766 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.298821 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 14:34:30.298939 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 14:34:30.299126 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16817,"thread_start_us":17264,"threads_started":1}
I20260810 14:34:30.300431 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.271483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.6.46.195:35153)
0810 14:34:30.272308 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:30.272317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:30.288058 (+ 15741us) server_negotiation.cc:407] Connection header received
0810 14:34:30.289105 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:30.289115 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:30.289372 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:30.289540 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:30.290598 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.292381 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.296447 (+ 4066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.297928 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.298056 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.298290 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:30.299916 (+ 1626us) server_negotiation.cc:299] Negotiation successful
0810 14:34:30.300091 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":416,"threads_started":1}
I20260810 14:34:30.306131 28582 tablet_service.cc:1467] Processing CreateTablet for tablet b2a3434e33584cc28ecc61ab473cb472 (DEFAULT_TABLE table=test-table [id=b0ba19dd83d04534a5a7b81286332803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:34:30.307722 28582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a3434e33584cc28ecc61ab473cb472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:30.311623 28730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.273960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:37575 (local address 127.0.0.1:50234)
0810 14:34:30.293247 (+ 19287us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:30.293265 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:30.293382 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:30.302105 (+ 8723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:30.302111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:30.302144 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:30.302585 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:30.302596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.305015 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.305020 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:30.308596 (+ 3576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:30.308606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.310825 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.310832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:30.311002 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.311049 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 14:34:30.311131 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 14:34:30.311259 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19151,"mutex_wait_us":1573,"thread_start_us":19013,"threads_started":1}
I20260810 14:34:30.314424 28727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.274909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50234 (local address 127.6.46.194:37575)
0810 14:34:30.294625 (+ 19716us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:30.294633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:30.294664 (+ 31us) server_negotiation.cc:407] Connection header received
0810 14:34:30.294790 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:30.294797 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:30.295014 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:30.295213 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:30.303389 (+ 8176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.304835 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.308763 (+ 3928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.310363 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.310495 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.310671 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:30.313930 (+ 3259us) server_negotiation.cc:299] Negotiation successful
0810 14:34:30.314138 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19564,"thread_start_us":19475,"threads_started":1}
I20260810 14:34:30.316510 28725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.274460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:35759 (local address 127.0.0.1:34330)
0810 14:34:30.303116 (+ 28656us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:30.303130 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:30.303238 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:30.304051 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:30.304055 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:30.304080 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:30.304379 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:30.304389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.306332 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.306337 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:30.308363 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:30.308379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.315668 (+ 7289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.315676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:30.315857 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.315891 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 14:34:30.315986 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 14:34:30.316138 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28550,"mutex_wait_us":1,"thread_start_us":28398,"threads_started":1}
I20260810 14:34:30.317209 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.275437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34330 (local address 127.6.46.193:35759)
0810 14:34:30.294241 (+ 18804us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:30.294250 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:30.303233 (+ 8983us) server_negotiation.cc:407] Connection header received
0810 14:34:30.303367 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:30.303374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:30.303600 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:30.303789 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:30.304593 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.306174 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.312039 (+ 5865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.313457 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.313599 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.313820 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:30.316715 (+ 2895us) server_negotiation.cc:299] Negotiation successful
0810 14:34:30.316877 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18638,"thread_start_us":18470,"threads_started":1}
I20260810 14:34:30.318166 28362 tablet_service.cc:1467] Processing CreateTablet for tablet b2a3434e33584cc28ecc61ab473cb472 (DEFAULT_TABLE table=test-table [id=b0ba19dd83d04534a5a7b81286332803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:34:30.319530 28362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a3434e33584cc28ecc61ab473cb472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:30.321803 28112 tablet_service.cc:1467] Processing CreateTablet for tablet b2a3434e33584cc28ecc61ab473cb472 (DEFAULT_TABLE table=test-table [id=b0ba19dd83d04534a5a7b81286332803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:34:30.324635 28112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2a3434e33584cc28ecc61ab473cb472. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:30.345223 28731 tablet_bootstrap.cc:492] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa: Bootstrap starting.
I20260810 14:34:30.349982 28731 tablet_bootstrap.cc:654] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:30.351514 28733 tablet_bootstrap.cc:492] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686: Bootstrap starting.
I20260810 14:34:30.354862 28732 tablet_bootstrap.cc:492] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8: Bootstrap starting.
I20260810 14:34:30.356433 28731 log.cc:824] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:30.361318 28732 tablet_bootstrap.cc:654] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:30.364920 28733 tablet_bootstrap.cc:654] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:30.367003 28732 log.cc:824] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:30.367640 28733 log.cc:824] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:30.373193 28733 tablet_bootstrap.cc:492] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686: No bootstrap required, opened a new log
I20260810 14:34:30.373521 28733 ts_tablet_manager.cc:1397] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.007s
I20260810 14:34:30.375658 28731 tablet_bootstrap.cc:492] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa: No bootstrap required, opened a new log
I20260810 14:34:30.376077 28731 ts_tablet_manager.cc:1397] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.005s
I20260810 14:34:30.383628 28732 tablet_bootstrap.cc:492] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8: No bootstrap required, opened a new log
I20260810 14:34:30.383970 28732 ts_tablet_manager.cc:1397] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.007s
I20260810 14:34:30.384931 28733 raft_consensus.cc:348] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513efccc57454ea89806c41e5d94bfaa" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35153 } } peers { permanent_uuid: "c06ed25bbe574ee98a8b2f316f8e9af8" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37575 } } peers { permanent_uuid: "404628b4b522431e9472df2ccf391686" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35759 } }
I20260810 14:34:30.385496 28733 raft_consensus.cc:374] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:30.385663 28733 raft_consensus.cc:729] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404628b4b522431e9472df2ccf391686, State: Initialized, Role: FOLLOWER
I20260810 14:34:30.386502 28733 consensus_queue.cc:260] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513efccc57454ea89806c41e5d94bfaa" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35153 } } peers { permanent_uuid: "c06ed25bbe574ee98a8b2f316f8e9af8" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37575 } } peers { permanent_uuid: "404628b4b522431e9472df2ccf391686" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35759 } }
I20260810 14:34:30.396477 28731 raft_consensus.cc:348] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513efccc57454ea89806c41e5d94bfaa" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35153 } } peers { permanent_uuid: "c06ed25bbe574ee98a8b2f316f8e9af8" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37575 } } peers { permanent_uuid: "404628b4b522431e9472df2ccf391686" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35759 } }
I20260810 14:34:30.401643 28731 raft_consensus.cc:374] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:30.401798 28731 raft_consensus.cc:729] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 513efccc57454ea89806c41e5d94bfaa, State: Initialized, Role: FOLLOWER
I20260810 14:34:30.402733 28229 heartbeater.cc:503] Master 127.6.46.254:43951 was elected leader, sending a full tablet report...
I20260810 14:34:30.403488 28733 ts_tablet_manager.cc:1428] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686: Time spent starting tablet: real 0.030s user 0.010s sys 0.018s
I20260810 14:34:30.403785 28732 raft_consensus.cc:348] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513efccc57454ea89806c41e5d94bfaa" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35153 } } peers { permanent_uuid: "c06ed25bbe574ee98a8b2f316f8e9af8" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37575 } } peers { permanent_uuid: "404628b4b522431e9472df2ccf391686" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35759 } }
I20260810 14:34:30.406376 28731 consensus_queue.cc:260] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513efccc57454ea89806c41e5d94bfaa" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35153 } } peers { permanent_uuid: "c06ed25bbe574ee98a8b2f316f8e9af8" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37575 } } peers { permanent_uuid: "404628b4b522431e9472df2ccf391686" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35759 } }
I20260810 14:34:30.407032 28732 raft_consensus.cc:374] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:30.407205 28732 raft_consensus.cc:729] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c06ed25bbe574ee98a8b2f316f8e9af8, State: Initialized, Role: FOLLOWER
I20260810 14:34:30.407917 28732 consensus_queue.cc:260] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513efccc57454ea89806c41e5d94bfaa" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35153 } } peers { permanent_uuid: "c06ed25bbe574ee98a8b2f316f8e9af8" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37575 } } peers { permanent_uuid: "404628b4b522431e9472df2ccf391686" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35759 } }
I20260810 14:34:30.410315 28471 heartbeater.cc:503] Master 127.6.46.254:43951 was elected leader, sending a full tablet report...
I20260810 14:34:30.410559 28731 ts_tablet_manager.cc:1428] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa: Time spent starting tablet: real 0.034s user 0.010s sys 0.002s
I20260810 14:34:30.411692 28732 ts_tablet_manager.cc:1428] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8: Time spent starting tablet: real 0.028s user 0.012s sys 0.000s
I20260810 14:34:30.415086 28691 heartbeater.cc:503] Master 127.6.46.254:43951 was elected leader, sending a full tablet report...
W20260810 14:34:30.471760 28472 tablet.cc:2369] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:34:30.534906 28230 tablet.cc:2369] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:34:30.632092 28694 tablet.cc:2369] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:34:30.700685 28741 raft_consensus.cc:482] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:34:30.700951 28741 raft_consensus.cc:504] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513efccc57454ea89806c41e5d94bfaa" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35153 } } peers { permanent_uuid: "c06ed25bbe574ee98a8b2f316f8e9af8" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37575 } } peers { permanent_uuid: "404628b4b522431e9472df2ccf391686" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35759 } }
I20260810 14:34:30.702553 28741 leader_election.cc:288] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c06ed25bbe574ee98a8b2f316f8e9af8 (127.6.46.194:37575), 404628b4b522431e9472df2ccf391686 (127.6.46.193:35759)
I20260810 14:34:30.711135 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.703092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:49257 (local address 127.6.46.193:35759)
0810 14:34:30.703307 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:30.703313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:30.704019 (+ 706us) server_negotiation.cc:407] Connection header received
0810 14:34:30.704142 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:30.704147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:30.704252 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:30.704353 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:30.705155 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.706257 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.708962 (+ 2705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.710182 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.710336 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.710405 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:30.710768 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 14:34:30.710890 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 14:34:30.711156 28727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.703079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:38155 (local address 127.6.46.194:37575)
0810 14:34:30.703307 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:30.703313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:30.703938 (+ 625us) server_negotiation.cc:407] Connection header received
0810 14:34:30.704158 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:30.704163 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:30.704250 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:30.704353 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:30.705140 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.706257 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.708970 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.710197 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.710333 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.710405 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:30.710768 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 14:34:30.710890 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 14:34:30.711421 28759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.702930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:35759 (local address 127.6.46.195:49257)
0810 14:34:30.703913 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:30.703927 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:30.704017 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:30.704411 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:30.704417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:30.704462 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:30.704995 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:30.705005 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.706393 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.706398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:30.708815 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:30.708831 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.710362 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.710372 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:30.710599 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.710657 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 14:34:30.710743 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 14:34:30.710950 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":884,"mutex_wait_us":199,"spinlock_wait_cycles":50816,"thread_start_us":529,"threads_started":1}
I20260810 14:34:30.711441 28757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.702889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:37575 (local address 127.6.46.195:38155)
0810 14:34:30.703818 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:30.703838 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:30.703990 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:30.704411 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:30.704417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:30.704461 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:30.704924 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:30.704942 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.706393 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.706398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:30.708815 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:30.708831 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.710384 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.710390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:30.710607 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.710648 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 14:34:30.710743 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 14:34:30.710950 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":271,"spinlock_wait_cycles":2304,"thread_start_us":554,"threads_started":1}
I20260810 14:34:30.712679 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a3434e33584cc28ecc61ab473cb472" candidate_uuid: "513efccc57454ea89806c41e5d94bfaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404628b4b522431e9472df2ccf391686" is_pre_election: true
I20260810 14:34:30.712680 28406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a3434e33584cc28ecc61ab473cb472" candidate_uuid: "513efccc57454ea89806c41e5d94bfaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06ed25bbe574ee98a8b2f316f8e9af8" is_pre_election: true
I20260810 14:34:30.713265 28406 raft_consensus.cc:2454] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 513efccc57454ea89806c41e5d94bfaa in term 0.
I20260810 14:34:30.713265 28150 raft_consensus.cc:2454] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 513efccc57454ea89806c41e5d94bfaa in term 0.
I20260810 14:34:30.713974 28512 leader_election.cc:302] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 513efccc57454ea89806c41e5d94bfaa, c06ed25bbe574ee98a8b2f316f8e9af8; no voters:
I20260810 14:34:30.714401 28741 raft_consensus.cc:2781] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:34:30.714530 28741 raft_consensus.cc:482] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:34:30.714622 28741 raft_consensus.cc:3037] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:30.716729 28741 raft_consensus.cc:504] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513efccc57454ea89806c41e5d94bfaa" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35153 } } peers { permanent_uuid: "c06ed25bbe574ee98a8b2f316f8e9af8" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37575 } } peers { permanent_uuid: "404628b4b522431e9472df2ccf391686" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35759 } }
I20260810 14:34:30.717258 28741 leader_election.cc:288] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [CANDIDATE]: Term 1 election: Requested vote from peers c06ed25bbe574ee98a8b2f316f8e9af8 (127.6.46.194:37575), 404628b4b522431e9472df2ccf391686 (127.6.46.193:35759)
I20260810 14:34:30.717737 28406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a3434e33584cc28ecc61ab473cb472" candidate_uuid: "513efccc57454ea89806c41e5d94bfaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c06ed25bbe574ee98a8b2f316f8e9af8"
I20260810 14:34:30.717736 28150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2a3434e33584cc28ecc61ab473cb472" candidate_uuid: "513efccc57454ea89806c41e5d94bfaa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "404628b4b522431e9472df2ccf391686"
I20260810 14:34:30.717939 28406 raft_consensus.cc:3037] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:30.717939 28150 raft_consensus.cc:3037] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:30.721069 28150 raft_consensus.cc:2454] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 513efccc57454ea89806c41e5d94bfaa in term 1.
I20260810 14:34:30.721096 28406 raft_consensus.cc:2454] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 513efccc57454ea89806c41e5d94bfaa in term 1.
I20260810 14:34:30.721455 28512 leader_election.cc:302] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 404628b4b522431e9472df2ccf391686, 513efccc57454ea89806c41e5d94bfaa, c06ed25bbe574ee98a8b2f316f8e9af8; no voters:
I20260810 14:34:30.721758 28741 raft_consensus.cc:2781] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:34:30.722206 28741 raft_consensus.cc:686] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [term 1 LEADER]: Becoming Leader. State: Replica: 513efccc57454ea89806c41e5d94bfaa, State: Running, Role: LEADER
I20260810 14:34:30.722694 28741 consensus_queue.cc:237] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513efccc57454ea89806c41e5d94bfaa" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35153 } } peers { permanent_uuid: "c06ed25bbe574ee98a8b2f316f8e9af8" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37575 } } peers { permanent_uuid: "404628b4b522431e9472df2ccf391686" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35759 } }
I20260810 14:34:30.726848 27885 catalog_manager.cc:5495] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa reported cstate change: term changed from 0 to 1, leader changed from <none> to 513efccc57454ea89806c41e5d94bfaa (127.6.46.195). New cstate: current_term: 1 leader_uuid: "513efccc57454ea89806c41e5d94bfaa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "513efccc57454ea89806c41e5d94bfaa" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 35153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c06ed25bbe574ee98a8b2f316f8e9af8" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "404628b4b522431e9472df2ccf391686" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35759 } health_report { overall_health: UNKNOWN } } }
I20260810 14:34:30.786440 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.780978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.6.46.195:35153)
0810 14:34:30.781213 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:30.781219 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:30.781480 (+ 261us) server_negotiation.cc:407] Connection header received
0810 14:34:30.781659 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:30.781663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:30.781743 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:30.781861 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:34:30.782230 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.782909 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.784408 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.784801 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.784895 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.785352 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:34:30.785841 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:34:30.785964 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:30.786128 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 14:34:30.786195 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:34:30.794765 28150 raft_consensus.cc:1261] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686 [term 1 FOLLOWER]: Refusing update from remote peer 513efccc57454ea89806c41e5d94bfaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:34:30.794765 28406 raft_consensus.cc:1261] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8 [term 1 FOLLOWER]: Refusing update from remote peer 513efccc57454ea89806c41e5d94bfaa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:34:30.795845 28761 consensus_queue.cc:1035] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c06ed25bbe574ee98a8b2f316f8e9af8" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:34:30.796216 28741 consensus_queue.cc:1035] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [LEADER]: Connected to new peer: Peer: permanent_uuid: "404628b4b522431e9472df2ccf391686" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:34:30.807705 28767 mvcc.cc:205] Tried to move back new op lower bound from 7316981640364175360 to 7316981640083664896. Current Snapshot: MvccSnapshot[applied={T|T < 7316981640364175360}]
I20260810 14:34:30.820228 27884 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:34:30.820449 27884 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:34:30.823580 27884 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c576a55b62784d878a86993e7a6ed601: Sending DeleteTablet for 3 replicas of tablet b2a3434e33584cc28ecc61ab473cb472
I20260810 14:34:30.824620 28362 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a3434e33584cc28ecc61ab473cb472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:34:30 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:50234
I20260810 14:34:30.824608 28112 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a3434e33584cc28ecc61ab473cb472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:34:30 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:34330
I20260810 14:34:30.824626 28582 tablet_service.cc:1514] Processing DeleteTablet for tablet b2a3434e33584cc28ecc61ab473cb472 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:34:30 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:33838
I20260810 14:34:30.826552 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 27953
I20260810 14:34:30.852137 28789 tablet_replica.cc:330] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa: stopping tablet replica
I20260810 14:34:30.852639 28789 raft_consensus.cc:2229] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:34:30.856534 28789 raft_consensus.cc:2258] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:34:30.857153 28787 tablet_replica.cc:330] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686: stopping tablet replica
I20260810 14:34:30.857537 28787 raft_consensus.cc:2229] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:34:30.858058 28787 raft_consensus.cc:2258] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:34:30.860285 28787 ts_tablet_manager.cc:1905] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:34:30.860350 28789 ts_tablet_manager.cc:1905] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:34:30.862452 28788 tablet_replica.cc:330] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8: stopping tablet replica
I20260810 14:34:30.862798 28788 raft_consensus.cc:2229] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:34:30.863281 28788 raft_consensus.cc:2258] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:34:30.865351 28788 ts_tablet_manager.cc:1905] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:34:30.869397 28789 ts_tablet_manager.cc:1918] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:34:30.869567 28789 log.cc:1196] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/b2a3434e33584cc28ecc61ab473cb472
I20260810 14:34:30.870150 28789 ts_tablet_manager.cc:1939] T b2a3434e33584cc28ecc61ab473cb472 P 513efccc57454ea89806c41e5d94bfaa: Deleting consensus metadata
I20260810 14:34:30.872188 28788 ts_tablet_manager.cc:1918] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:34:30.872397 28788 log.cc:1196] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/b2a3434e33584cc28ecc61ab473cb472
I20260810 14:34:30.872375 27857 catalog_manager.cc:4841] TS 513efccc57454ea89806c41e5d94bfaa (127.6.46.195:35153): tablet b2a3434e33584cc28ecc61ab473cb472 (table test-table [id=b0ba19dd83d04534a5a7b81286332803]) successfully deleted
I20260810 14:34:30.872823 28788 ts_tablet_manager.cc:1939] T b2a3434e33584cc28ecc61ab473cb472 P c06ed25bbe574ee98a8b2f316f8e9af8: Deleting consensus metadata
I20260810 14:34:30.874274 27860 catalog_manager.cc:4841] TS c06ed25bbe574ee98a8b2f316f8e9af8 (127.6.46.194:37575): tablet b2a3434e33584cc28ecc61ab473cb472 (table test-table [id=b0ba19dd83d04534a5a7b81286332803]) successfully deleted
I20260810 14:34:30.876233 28787 ts_tablet_manager.cc:1918] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:34:30.876399 28787 log.cc:1196] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/b2a3434e33584cc28ecc61ab473cb472
I20260810 14:34:30.876943 28787 ts_tablet_manager.cc:1939] T b2a3434e33584cc28ecc61ab473cb472 P 404628b4b522431e9472df2ccf391686: Deleting consensus metadata
I20260810 14:34:30.878784 27855 catalog_manager.cc:4841] TS 404628b4b522431e9472df2ccf391686 (127.6.46.193:35759): tablet b2a3434e33584cc28ecc61ab473cb472 (table test-table [id=b0ba19dd83d04534a5a7b81286332803]) successfully deleted
I20260810 14:34:30.878945 28728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.827525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.6.46.193:35759)
0810 14:34:30.827713 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:30.827720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:30.849122 (+ 21402us) server_negotiation.cc:407] Connection header received
0810 14:34:30.849828 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:30.849834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:30.849928 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:30.850098 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:30.852920 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.854138 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.860965 (+ 6827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.861622 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.861767 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.871273 (+ 9506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:30.871287 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:30.871304 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:30.871345 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:30.875555 (+ 4210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:30.876367 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:30.876377 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:30.876380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:30.876618 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:30.877016 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:30.877021 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:30.877024 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:30.877389 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:30.877737 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:30.877980 (+ 243us) server_negotiation.cc:299] Negotiation successful
0810 14:34:30.878251 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:34:30.879565 28218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:30.966395 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 27953
I20260810 14:34:30.990075 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 28237
I20260810 14:34:31.003252 28727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:30.990902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.6.46.194:37575)
0810 14:34:30.991141 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:30.991147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:30.991167 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:34:30.991333 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:30.991338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:30.991407 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:30.991504 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:30.992844 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.993526 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.994852 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:30.995264 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:30.995377 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:30.997808 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:30.997823 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:30.997838 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:30.997871 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:31.000521 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:31.001164 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:31.001172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:31.001174 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:31.001316 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:31.001694 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:31.001698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:31.001700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:31.002100 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:31.002379 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:31.002709 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 14:34:31.002918 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260810 14:34:31.003822 28464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:31.098755 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 28237
I20260810 14:34:31.126371 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 28478
I20260810 14:34:31.142690 28722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:31.127135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33868 (local address 127.6.46.195:35153)
0810 14:34:31.127323 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:31.127329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:31.127490 (+ 161us) server_negotiation.cc:407] Connection header received
0810 14:34:31.127682 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:31.127688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:31.127772 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:31.127867 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:31.129702 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:31.130893 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:31.132873 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:31.133384 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:31.133484 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:31.136782 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:31.136793 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:31.136809 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:31.136839 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:31.139776 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:31.140598 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:31.140605 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:31.140607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:31.140764 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:31.141194 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:31.141198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:31.141199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:31.141535 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:31.141778 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:31.142213 (+ 435us) server_negotiation.cc:299] Negotiation successful
0810 14:34:31.142380 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:34:31.143299 28686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:31.264300 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 28478
I20260810 14:34:31.290927 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 27829
I20260810 14:34:31.292099 27940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:31.402366 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 27829
I20260810 14:34:31.433604 6331 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 14:34:31.777994 6331 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 14:34:31.778256 6331 mini_ranger.cc:66] Stopping Ranger...
I20260810 14:34:32.122332 6331 mini_ranger.cc:68] Stopped Ranger
2026-08-10 14:34:32.122 UTC [26295] LOG: received smart shutdown request
2026-08-10 14:34:32.125 UTC [26295] LOG: background worker "logical replication launcher" (PID 26307) exited with exit code 1
2026-08-10 14:34:32.125 UTC [26302] LOG: shutting down
2026-08-10 14:34:32.245 UTC [26295] LOG: database system is shut down
2026-08-10T14:34:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50834 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:34:32 dist-test-slave-q9h9 krb5kdc[28817](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:34:32 dist-test-slave-q9h9 krb5kdc[28817](info): set up 2 sockets
Aug 10 14:34:32 dist-test-slave-q9h9 krb5kdc[28817](info): commencing operation
krb5kdc: starting...
W20260810 14:34:33.585915 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:34:33 dist-test-slave-q9h9 krb5kdc[28817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:34:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:34:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:34:33.821878 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:46023
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:46023,127.6.46.253:39717,127.6.46.252:37173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:34:34.142839 28837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:34.143181 28837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:34.143321 28837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:34.150321 28837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:34:34.150432 28837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:34.150512 28837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:34.150586 28837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:34:34.150647 28837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:34:34.160234 28837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:46023,127.6.46.253:39717,127.6.46.252:37173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:46023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:34.161535 28837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:34.162823 28837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:34.174127 28847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:34.174119 28851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:34.174165 28849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:34.175001 28837 server_base.cc:1034] running on GCE node
I20260810 14:34:34.175773 28837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:34.177109 28837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:34.178313 28837 hybrid_clock.cc:648] HybridClock initialized: now 1786372474178260 us; error 47 us; skew 500 ppm
I20260810 14:34:34.178634 28837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:34 dist-test-slave-q9h9 krb5kdc[28817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:34.181964 28837 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:34:34.183377 28837 webserver.cc:460] Webserver started at http://127.6.46.254:44369/ using document root <none> and password file <none>
I20260810 14:34:34.184028 28837 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:34.184152 28837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:34.184429 28837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:34:34.187354 28837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "b07b596943c848a8acbe0da9984ab33d"
format_stamp: "Formatted at 2026-08-10 14:34:34 on dist-test-slave-q9h9"
server_key: "8268df569c706d2e3f430cd297a6bc60"
server_key_iv: "1dc6b57cab34c15367c98e4876d86a50"
server_key_version: "encryptionkey@0"
I20260810 14:34:34.188115 28837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "b07b596943c848a8acbe0da9984ab33d"
format_stamp: "Formatted at 2026-08-10 14:34:34 on dist-test-slave-q9h9"
server_key: "8268df569c706d2e3f430cd297a6bc60"
server_key_iv: "1dc6b57cab34c15367c98e4876d86a50"
server_key_version: "encryptionkey@0"
I20260810 14:34:34.192951 28837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:34:34.196152 28861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:34.197351 28837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 14:34:34.197503 28837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "b07b596943c848a8acbe0da9984ab33d"
format_stamp: "Formatted at 2026-08-10 14:34:34 on dist-test-slave-q9h9"
server_key: "8268df569c706d2e3f430cd297a6bc60"
server_key_iv: "1dc6b57cab34c15367c98e4876d86a50"
server_key_version: "encryptionkey@0"
I20260810 14:34:34.197665 28837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:34.217108 28837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:34.220712 28837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:34.220952 28837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:34.238600 28837 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:46023
I20260810 14:34:34.238655 28955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:46023 every 8 connection(s)
I20260810 14:34:34.239830 28837 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:34:34.243530 28956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:34.244669 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 28837
I20260810 14:34:34.244858 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:34:34.245474 6331 external_mini_cluster.cc:1428] Setting key a842f57cb65a4704156926f8bd8c964a
I20260810 14:34:34.249794 28956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:34:34 dist-test-slave-q9h9 krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.253@KRBTEST.COM" created.
I20260810 14:34:34.275276 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.252717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.6.46.254:46023)
0810 14:34:34.253471 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:34.253485 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:34.253911 (+ 426us) server_negotiation.cc:407] Connection header received
0810 14:34:34.255901 (+ 1990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:34.255907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:34.256114 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:34.256271 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:34.257493 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.259268 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.260592 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.261436 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.261619 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:34.268482 (+ 6863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:34.268494 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:34.268510 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:34.268553 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:34.271453 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:34.272114 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:34.272120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:34.272122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:34.272305 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:34.272694 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:34.272700 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:34.272703 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:34.273331 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:34.273530 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:34.274058 (+ 528us) server_negotiation.cc:299] Negotiation successful
0810 14:34:34.274413 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":543,"thread_start_us":350,"threads_started":1}
I20260810 14:34:34.275256 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.252718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:46023 (local address 127.0.0.1:45394)
0810 14:34:34.253760 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.253799 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:34.255514 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:34.256339 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:34.256347 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:34.256972 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:34.257296 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:34.257312 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.259411 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.259414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:34.260452 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:34.260464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.261644 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.261649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:34.261769 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:34.265915 (+ 4146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:34.265936 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:34.268247 (+ 2311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:34.271575 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:34.271585 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:34.271593 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:34.272008 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:34.272421 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:34.272424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:34.272426 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:34.272580 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:34.273466 (+ 886us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:34.273472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:34.273665 (+ 193us) client_negotiation.cc:769] Sending connection context
0810 14:34:34.273860 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 14:34:34.274251 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"spinlock_wait_cycles":16768,"thread_start_us":658,"threads_started":1}
I20260810 14:34:34.278081 28956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:34:34.279175 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.278754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:39717 (local address 127.0.0.1:48186)
0810 14:34:34.278920 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.279047 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 14:34:34.279440 28868 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
I20260810 14:34:34.280893 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.280535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:39717 (local address 127.0.0.1:48200)
0810 14:34:34.280686 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.280761 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 14:34:34.281080 28956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:39717: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.253.keytab.
Entry for principal kudu/127.6.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.253.keytab.
I20260810 14:34:34.299297 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.253:39717
--webserver_interface=127.6.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.253.keytab
--principal=kudu/127.6.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:46023,127.6.46.253:39717,127.6.46.252:37173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:34:34.330422 28956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } attempt: 1
I20260810 14:34:34.331512 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.331094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:39717 (local address 127.0.0.1:48208)
0810 14:34:34.331276 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.331376 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 14:34:34.332480 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.332131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:39717 (local address 127.0.0.1:48214)
0810 14:34:34.332285 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.332352 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 14:34:34.332665 28956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:39717: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
I20260810 14:34:34.400873 28956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } attempt: 2
I20260810 14:34:34.401975 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.401509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:39717 (local address 127.0.0.1:48226)
0810 14:34:34.401740 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.401833 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 14:34:34.403146 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.402713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:39717 (local address 127.0.0.1:48240)
0810 14:34:34.402866 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.402976 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 14:34:34.403344 28956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:39717: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
I20260810 14:34:34.494637 28956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } attempt: 3
I20260810 14:34:34.495962 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.495389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:39717 (local address 127.0.0.1:48252)
0810 14:34:34.495651 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.495777 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 14:34:34.497076 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.496624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:39717 (local address 127.0.0.1:48262)
0810 14:34:34.496796 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.496906 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 14:34:34.497279 28956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:39717: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
W20260810 14:34:34.611953 28970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:34.612202 28970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:34.612294 28970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:34.619109 28970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:34:34.619202 28970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:34.619262 28970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:34.619336 28970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:34:34.619391 28970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:34:34.628672 28970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:46023,127.6.46.253:39717,127.6.46.252:37173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.253.keytab
--principal=kudu/127.6.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.253:39717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:34.629784 28970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:34.630972 28970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:34:34.640628 28956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } attempt: 4
W20260810 14:34:34.641728 28980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:34.641944 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.641216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:39717 (local address 127.0.0.1:48276)
0810 14:34:34.641678 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.641783 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287}
W20260810 14:34:34.641755 28983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:34.642625 28970 server_base.cc:1034] running on GCE node
W20260810 14:34:34.642808 28981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:34.643270 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.642887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:39717 (local address 127.0.0.1:48286)
0810 14:34:34.643043 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.643144 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 14:34:34.643460 28956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:39717: Network error: Client connection negotiation failed: client connection to 127.6.46.253:39717: connect: Connection refused (error 111)
I20260810 14:34:34.643687 28970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:34.645073 28970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:34.646277 28970 hybrid_clock.cc:648] HybridClock initialized: now 1786372474646239 us; error 41 us; skew 500 ppm
I20260810 14:34:34.646610 28970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:34 dist-test-slave-q9h9 krb5kdc[28817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:34.649577 28970 init.cc:377] Logged in from keytab as kudu/127.6.46.253@KRBTEST.COM (short username kudu)
I20260810 14:34:34.651021 28970 webserver.cc:460] Webserver started at http://127.6.46.253:41813/ using document root <none> and password file <none>
I20260810 14:34:34.651666 28970 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:34.651782 28970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:34.652071 28970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:34:34.654917 28970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/data/instance:
uuid: "77701590d5bd4af08a782d9ef98fd569"
format_stamp: "Formatted at 2026-08-10 14:34:34 on dist-test-slave-q9h9"
server_key: "4a8bc5f927c77f63abd860cdfc7eba85"
server_key_iv: "9161151d8a106e208b916e2f3fdfe452"
server_key_version: "encryptionkey@0"
I20260810 14:34:34.655679 28970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/wal/instance:
uuid: "77701590d5bd4af08a782d9ef98fd569"
format_stamp: "Formatted at 2026-08-10 14:34:34 on dist-test-slave-q9h9"
server_key: "4a8bc5f927c77f63abd860cdfc7eba85"
server_key_iv: "9161151d8a106e208b916e2f3fdfe452"
server_key_version: "encryptionkey@0"
I20260810 14:34:34.660499 28970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:34:34.663715 28994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:34.664892 28970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:34:34.665046 28970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/wal
uuid: "77701590d5bd4af08a782d9ef98fd569"
format_stamp: "Formatted at 2026-08-10 14:34:34 on dist-test-slave-q9h9"
server_key: "4a8bc5f927c77f63abd860cdfc7eba85"
server_key_iv: "9161151d8a106e208b916e2f3fdfe452"
server_key_version: "encryptionkey@0"
I20260810 14:34:34.665222 28970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:34.679180 28970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:34.682726 28970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:34.682965 28970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:34.699466 28970 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.253:39717
I20260810 14:34:34.699517 29071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.253:39717 every 8 connection(s)
I20260810 14:34:34.700677 28970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/data/info.pb
I20260810 14:34:34.704161 29073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:34.709270 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 28970
I20260810 14:34:34.709455 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-1/wal/instance
I20260810 14:34:34.709964 6331 external_mini_cluster.cc:1428] Setting key 60a1efd30ded554981f24ae7d65490af
I20260810 14:34:34.710695 29073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:34:34 dist-test-slave-q9h9 krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.253@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:34.729854 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.713547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.6.46.254:46023)
0810 14:34:34.713747 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:34.713753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:34.714829 (+ 1076us) server_negotiation.cc:407] Connection header received
0810 14:34:34.716569 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:34.716575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:34.716656 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:34.716765 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:34.718117 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.719013 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.720443 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.720904 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.721002 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:34.724550 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:34.724554 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:34.724557 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:34.724591 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:34.727057 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:34.727742 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:34.727746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:34.727748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:34.727865 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:34.728254 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:34.728258 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:34.728260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:34.728547 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:34.728679 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:34.729326 (+ 647us) server_negotiation.cc:299] Negotiation successful
0810 14:34:34.729549 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 14:34:34.730759 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.713890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:46023 (local address 127.0.0.1:45400)
0810 14:34:34.714644 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.714697 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:34.716314 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:34.716932 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:34.716939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:34.717412 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:34.717936 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:34.717951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.719149 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.719152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:34.720304 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:34.720316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.721129 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.721136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:34.721391 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:34.722011 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:34.722083 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:34.724349 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:34.727184 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:34.727193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:34.727207 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:34.727621 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:34.727966 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:34.727971 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:34.727975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:34.728142 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:34.728677 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:34.728684 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:34.728965 (+ 281us) client_negotiation.cc:769] Sending connection context
0810 14:34:34.729184 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 14:34:34.729646 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":339,"threads_started":1}
I20260810 14:34:34.732328 29073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.46.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.252@KRBTEST.COM" created.
Aug 10 14:34:34 dist-test-slave-q9h9 krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.253@KRBTEST.COM for kudu/127.6.46.253@KRBTEST.COM
I20260810 14:34:34.746572 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.733044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:39717 (local address 127.0.0.1:48292)
0810 14:34:34.733255 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.733273 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:34.733372 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:34.734392 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:34.734396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:34.734834 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:34.734980 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:34.734990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.736365 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.736369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:34.737437 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:34.737451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.738296 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.738306 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:34.738475 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:34.738967 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:34.738988 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:34.741188 (+ 2200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:34.744076 (+ 2888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:34.744080 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:34.744083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:34.744538 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:34.744895 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:34.744899 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:34.744903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:34.745069 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:34.745681 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:34.745685 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:34.745806 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 14:34:34.745923 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 14:34:34.746205 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 14:34:34.746693 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.733283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.6.46.253:39717)
0810 14:34:34.733958 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:34.733963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:34.733988 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:34:34.734071 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:34.734079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:34.734253 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:34.734420 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:34.735140 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.736227 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.737592 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.738138 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.738229 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:34.741385 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:34.741395 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:34.741407 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:34.741435 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:34.743953 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:34.744666 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:34.744673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:34.744675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:34.744788 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:34.745183 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:34.745187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:34.745194 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:34.745534 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:34.745714 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:34.746106 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 14:34:34.746311 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":394,"threads_started":1}
I20260810 14:34:34.748765 29073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:34:34.749842 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.749415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:44996)
0810 14:34:34.749590 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.749707 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 14:34:34.750136 28998 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
I20260810 14:34:34.751581 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.751284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45006)
0810 14:34:34.751410 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.751486 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 14:34:34.751791 29073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:37173: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.252.keytab.
Entry for principal kudu/127.6.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.252.keytab.
I20260810 14:34:34.765290 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.252:37173
--webserver_interface=127.6.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.252.keytab
--principal=kudu/127.6.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:39149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:46023,127.6.46.253:39717,127.6.46.252:37173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:34:34.801077 29073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } attempt: 1
I20260810 14:34:34.802170 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.801676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45012)
0810 14:34:34.801854 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.801977 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 14:34:34.803179 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.802793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45020)
0810 14:34:34.802970 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.803055 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 14:34:34.803443 29073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:37173: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
I20260810 14:34:34.871662 29073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } attempt: 2
I20260810 14:34:34.872859 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.872383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45032)
0810 14:34:34.872580 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.872684 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 14:34:34.873823 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.873514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45046)
0810 14:34:34.873635 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.873703 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 14:34:34.874066 29073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:37173: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
I20260810 14:34:34.942726 28956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } attempt: 5
Aug 10 14:34:34 dist-test-slave-q9h9 krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for kudu/127.6.46.253@KRBTEST.COM
I20260810 14:34:34.956895 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.943454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:39717 (local address 127.0.0.1:48298)
0810 14:34:34.943676 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.943698 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:34.943830 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:34.944246 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:34.944253 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:34.944780 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:34.945017 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:34.945030 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.946073 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.946079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:34.947450 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:34.947469 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.948241 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.948248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:34.948433 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:34.949127 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:34.949152 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:34.951683 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:34.954576 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:34.954582 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:34.954587 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:34.954953 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:34.955307 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:34.955313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:34.955317 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:34.955455 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:34.955961 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:34.955966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:34.956120 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 14:34:34.956260 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 14:34:34.956528 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 14:34:34.956894 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.943593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48298 (local address 127.6.46.253:39717)
0810 14:34:34.943758 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:34.943762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:34.943782 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:34:34.943955 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:34.943958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:34.944055 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:34.944141 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:34.945209 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.945898 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.947614 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:34.948085 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:34.948178 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:34.951871 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:34.951876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:34.951879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:34.951913 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:34.954429 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:34.955065 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:34.955069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:34.955073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:34.955199 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:34.955555 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:34.955559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:34.955562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:34.955845 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:34.955977 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:34.956381 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 14:34:34.956599 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:34:34.957888 28956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:34:34.958947 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.958575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45048)
0810 14:34:34.958736 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.958815 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 14:34:34.959991 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.959671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45056)
0810 14:34:34.959807 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.959894 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 14:34:34.960201 28956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:37173: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
I20260810 14:34:34.965271 29073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } attempt: 3
I20260810 14:34:34.966169 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.965757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45070)
0810 14:34:34.965927 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.966006 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 14:34:34.967226 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:34.966835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45086)
0810 14:34:34.967049 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:34.967124 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
W20260810 14:34:34.967442 29073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:37173: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
I20260810 14:34:35.011394 28956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } attempt: 1
I20260810 14:34:35.012452 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.012031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45090)
0810 14:34:35.012214 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:35.012300 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 14:34:35.013370 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.013046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45098)
0810 14:34:35.013195 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:35.013269 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 14:34:35.013571 28956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:37173: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
I20260810 14:34:35.081764 28956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } attempt: 2
I20260810 14:34:35.082832 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.082368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45102)
0810 14:34:35.082595 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:35.082680 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 14:34:35.083750 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.083422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45118)
0810 14:34:35.083574 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:35.083638 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 14:34:35.083951 28956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:37173: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
W20260810 14:34:35.084599 29085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:35.084851 29085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:35.084972 29085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:35.091946 29085 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:34:35.092049 29085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:35.092129 29085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:35.092206 29085 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:34:35.092264 29085 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:34:35.101626 29085 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:39149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:46023,127.6.46.253:39717,127.6.46.252:37173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.252.keytab
--principal=kudu/127.6.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.252:37173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:35.102825 29085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:35.104019 29085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 14:34:35.110613 29073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } attempt: 4
I20260810 14:34:35.111618 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.111202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45130)
0810 14:34:35.111397 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:35.111478 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 14:34:35.112535 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.112265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45134)
0810 14:34:35.112380 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:35.112442 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 14:34:35.112735 29073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:37173: Network error: Client connection negotiation failed: client connection to 127.6.46.252:37173: connect: Connection refused (error 111)
W20260810 14:34:35.115438 29092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:35.115466 29095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:35.115393 29093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:35.116396 29085 server_base.cc:1034] running on GCE node
I20260810 14:34:35.117079 29085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:35.118342 29085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:35.119534 29085 hybrid_clock.cc:648] HybridClock initialized: now 1786372475119501 us; error 26 us; skew 500 ppm
I20260810 14:34:35.119832 29085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:35 dist-test-slave-q9h9 krb5kdc[28817](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372475, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:35.122685 29085 init.cc:377] Logged in from keytab as kudu/127.6.46.252@KRBTEST.COM (short username kudu)
I20260810 14:34:35.124020 29085 webserver.cc:460] Webserver started at http://127.6.46.252:41687/ using document root <none> and password file <none>
I20260810 14:34:35.124639 29085 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:35.124752 29085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:35.125022 29085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:34:35.127830 29085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/data/instance:
uuid: "6eefaec2049c475ab4f62c5b6c59fa0a"
format_stamp: "Formatted at 2026-08-10 14:34:35 on dist-test-slave-q9h9"
server_key: "b395bc97019c06f9d4999df95f847b63"
server_key_iv: "09c7e1a198f4a87d419276a20ff37f28"
server_key_version: "encryptionkey@0"
I20260810 14:34:35.128572 29085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/wal/instance:
uuid: "6eefaec2049c475ab4f62c5b6c59fa0a"
format_stamp: "Formatted at 2026-08-10 14:34:35 on dist-test-slave-q9h9"
server_key: "b395bc97019c06f9d4999df95f847b63"
server_key_iv: "09c7e1a198f4a87d419276a20ff37f28"
server_key_version: "encryptionkey@0"
I20260810 14:34:35.133273 29085 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:34:35.136524 29108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:35.137786 29085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:34:35.137975 29085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/wal
uuid: "6eefaec2049c475ab4f62c5b6c59fa0a"
format_stamp: "Formatted at 2026-08-10 14:34:35 on dist-test-slave-q9h9"
server_key: "b395bc97019c06f9d4999df95f847b63"
server_key_iv: "09c7e1a198f4a87d419276a20ff37f28"
server_key_version: "encryptionkey@0"
I20260810 14:34:35.138235 29085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:35.167827 29085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:35.171754 29085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:35.172031 29085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:35.190135 29085 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.252:37173
I20260810 14:34:35.190138 28956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } attempt: 3
I20260810 14:34:35.190151 29191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.252:37173 every 8 connection(s)
I20260810 14:34:35.191767 29085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/data/info.pb
I20260810 14:34:35.195763 29193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:35.197041 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 29085
I20260810 14:34:35.197261 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786372275129649-6331-0/minicluster-data/master-2/wal/instance
I20260810 14:34:35.197842 6331 external_mini_cluster.cc:1428] Setting key 99bf96bd2bb62cd3feb3b7d375ae5149
Aug 10 14:34:35 dist-test-slave-q9h9 krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for kudu/127.6.46.252@KRBTEST.COM
I20260810 14:34:35.206367 29193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:34:35.211993 28961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.190802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45136)
0810 14:34:35.191045 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:35.191069 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:35.191213 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:35.195234 (+ 4021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:35.195239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:35.195752 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:35.195956 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:35.195970 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.197830 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.197837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:35.199098 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:35.199119 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.200667 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.200675 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:35.200869 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.201553 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:35.201580 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:35.204835 (+ 3255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:35.208773 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:35.208779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:35.208783 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:35.209295 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:35.209852 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:35.209858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:35.209862 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:35.210046 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:35.210991 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:35.210996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:35.211186 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 14:34:35.211297 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 14:34:35.211569 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 14:34:35.213280 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.191608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.6.46.252:37173)
0810 14:34:35.192450 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:35.192468 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:35.192530 (+ 62us) server_negotiation.cc:407] Connection header received
0810 14:34:35.194459 (+ 1929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:35.194483 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:35.194872 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:35.195195 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:35.196136 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.197601 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.199301 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.200452 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.201283 (+ 831us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.205109 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:35.205121 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:35.205139 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:35.205184 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:35.208634 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.209433 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.209441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.209448 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.209694 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.210177 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.210181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.210185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.210850 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:35.211036 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:35.211607 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 14:34:35.211993 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":338,"threads_started":1}
Aug 10 14:34:35 dist-test-slave-q9h9 krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.252@KRBTEST.COM
Aug 10 14:34:35 dist-test-slave-q9h9 krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372475, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.252@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:35.220477 28956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d: Bootstrap starting.
I20260810 14:34:35.222643 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.198780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45150 (local address 127.6.46.252:37173)
0810 14:34:35.203690 (+ 4910us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:35.203695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:35.204033 (+ 338us) server_negotiation.cc:407] Connection header received
0810 14:34:35.204204 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:35.204207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:35.204258 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:35.204348 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:35.206062 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.207362 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.209269 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.210004 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.210136 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.216604 (+ 6468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:35.216611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:35.216615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:35.216651 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:35.219544 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.220555 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.220561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.220564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.220709 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.221132 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.221137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.221141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.221450 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:35.221747 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:35.222069 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 14:34:35.222265 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4768,"thread_start_us":4573,"threads_started":1}
I20260810 14:34:35.224361 28956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:35.224522 29197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.209106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:46023 (local address 127.0.0.1:45416)
0810 14:34:35.209909 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:35.209950 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:35.210203 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:35.210520 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:35.210527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:35.211217 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:35.211486 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:35.211494 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.212631 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.212635 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:35.213802 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:35.213814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.214776 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.214783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:35.214925 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.215470 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:35.215491 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:35.218180 (+ 2689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:35.221628 (+ 3448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:35.221639 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:35.221646 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:35.222144 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:35.222602 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:35.222606 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:35.222609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:35.222799 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:35.223526 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:35.223532 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:35.223692 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 14:34:35.223828 (+ 136us) client_negotiation.cc:240] Negotiation successful
0810 14:34:35.224114 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":421,"threads_started":1}
I20260810 14:34:35.224594 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.209080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.6.46.254:46023)
0810 14:34:35.209250 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:35.209256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:35.210075 (+ 819us) server_negotiation.cc:407] Connection header received
0810 14:34:35.210313 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:35.210318 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:35.210383 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:35.210461 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:35.211621 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.212465 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.213931 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.214583 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.214688 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.218371 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:35.218377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:35.218382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:35.218420 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:35.221479 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.222274 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.222279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.222283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.222450 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.222936 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.222941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.222944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.223315 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:35.223435 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:35.223969 (+ 534us) server_negotiation.cc:299] Negotiation successful
0810 14:34:35.224157 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:34:35.225805 28956 log.cc:824] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:35.225919 29193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:34:35.228791 28956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d: No bootstrap required, opened a new log
Aug 10 14:34:35 dist-test-slave-q9h9 krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
Aug 10 14:34:35 dist-test-slave-q9h9 krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372475, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.252@KRBTEST.COM for kudu/127.6.46.253@KRBTEST.COM
I20260810 14:34:35.238660 28956 raft_consensus.cc:348] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } }
I20260810 14:34:35.239063 28956 raft_consensus.cc:374] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:35.239195 28956 raft_consensus.cc:729] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b07b596943c848a8acbe0da9984ab33d, State: Initialized, Role: FOLLOWER
I20260810 14:34:35.239848 28956 consensus_queue.cc:260] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } }
I20260810 14:34:35.240624 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.226864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.6.46.253:39717)
0810 14:34:35.227042 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:35.227047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:35.227066 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:34:35.227122 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:35.227126 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:35.227213 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:35.227322 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:35.228283 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.229288 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.230868 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.231502 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.231622 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.234987 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:35.234991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:35.234994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:35.235025 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:35.237648 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.238461 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.238466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.238470 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.238625 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.239080 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.239086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.239089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.239407 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:35.239539 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:35.240069 (+ 530us) server_negotiation.cc:299] Negotiation successful
0810 14:34:35.240294 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:34:35.241501 29197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.226526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:39717 (local address 127.0.0.1:48312)
0810 14:34:35.226784 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:35.226800 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:35.226885 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:35.227360 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:35.227365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:35.227950 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:35.228119 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:35.228130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.229445 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.229451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:35.230719 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:35.230733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.231665 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.231672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:35.231862 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.232410 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:35.232431 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:35.234823 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:35.237763 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:35.237768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:35.237772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:35.238318 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:35.238777 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:35.238782 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:35.238785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:35.238949 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:35.239621 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:35.239626 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:35.239781 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 14:34:35.240866 (+ 1085us) client_negotiation.cc:240] Negotiation successful
0810 14:34:35.241177 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260810 14:34:35.242671 29193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:34:35.243368 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.224487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.6.46.254:46023)
0810 14:34:35.224805 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:35.224810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:35.225553 (+ 743us) server_negotiation.cc:407] Connection header received
0810 14:34:35.225708 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:35.225713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:35.225784 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:35.225879 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:35.227459 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.228286 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.229881 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.230427 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.230544 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.234755 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:35.234762 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:35.234767 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:35.234803 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:35.238290 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.239518 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.239522 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.239525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.239681 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.240220 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.240231 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.240234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.240668 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:35.241157 (+ 489us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:35.242631 (+ 1474us) server_negotiation.cc:299] Negotiation successful
0810 14:34:35.242990 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":740,"threads_started":1}
I20260810 14:34:35.243655 29202 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } } }
I20260810 14:34:35.244098 29202 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:34:35.245038 28956 sys_catalog.cc:564] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 14:34:35 dist-test-slave-q9h9 krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372475, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.252@KRBTEST.COM for kudu/127.6.46.252@KRBTEST.COM
Aug 10 14:34:35 dist-test-slave-q9h9 krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.253@KRBTEST.COM
I20260810 14:34:35.258780 29197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.243433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45158)
0810 14:34:35.244286 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:35.244301 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:35.244381 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:35.245212 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:35.245216 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:35.245621 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:35.245797 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:35.245807 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.246891 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.246895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:35.247919 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:35.247935 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.248947 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.248952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:35.249124 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.249683 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:35.249706 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:35.252454 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:35.256093 (+ 3639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:35.256098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:35.256102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:35.256568 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:35.257005 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:35.257010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:35.257014 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:35.257173 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:35.257812 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:35.257817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:35.257962 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 14:34:35.258151 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 14:34:35.258431 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748}
I20260810 14:34:35.259037 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.243721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45158 (local address 127.6.46.252:37173)
0810 14:34:35.243894 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:35.243900 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:35.244668 (+ 768us) server_negotiation.cc:407] Connection header received
0810 14:34:35.244721 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:35.244726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:35.244805 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:35.244891 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:35.245932 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.246765 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.248068 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.248666 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.248775 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.252647 (+ 3872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:35.252653 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:35.252658 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:35.252692 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:35.255955 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.256729 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.256735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.256738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.256881 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.257288 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.257293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.257296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.257679 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:35.257831 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:35.258505 (+ 674us) server_negotiation.cc:299] Negotiation successful
0810 14:34:35.258713 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:34:35.264619 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.245887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.6.46.253:39717)
0810 14:34:35.246043 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:35.246048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:35.246268 (+ 220us) server_negotiation.cc:407] Connection header received
0810 14:34:35.246462 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:35.246467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:35.246530 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:35.246585 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:35.248195 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.249212 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.251250 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.251878 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.251969 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.257630 (+ 5661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:35.257636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:35.257641 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:35.257670 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:35.261088 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.261890 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.261895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.261898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.262094 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.262544 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.262548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.262551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.262874 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:35.263148 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:35.264107 (+ 959us) server_negotiation.cc:299] Negotiation successful
0810 14:34:35.264316 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:34:35.264703 29193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a: Bootstrap starting.
I20260810 14:34:35.268921 29193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:35.270324 29193 log.cc:824] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:35.270808 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:34:35.264430 28837 master_runner.cc:186] Error getting master registration for 127.6.46.253:39717: OK, instance_id { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" instance_seqno: 1786372474694739 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:34:35.273094 28837 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:34:35.273375 29193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a: No bootstrap required, opened a new log
W20260810 14:34:35.282498 29216 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:34:35.282632 29216 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:34:35.283711 29193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } }
I20260810 14:34:35.284137 29193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:35.284252 29193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6eefaec2049c475ab4f62c5b6c59fa0a, State: Initialized, Role: FOLLOWER
I20260810 14:34:35.284945 29193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } }
I20260810 14:34:35.291802 29234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } } }
I20260810 14:34:35.292210 29234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:34:35.293913 29193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:34:35.304162 28960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.276723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.6.46.254:46023)
0810 14:34:35.276897 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:35.276903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:35.286546 (+ 9643us) server_negotiation.cc:407] Connection header received
0810 14:34:35.286717 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:35.286722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:35.286795 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:35.286883 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:35.289258 (+ 2375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.290414 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.292578 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.293318 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.293415 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.297037 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:35.297044 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:35.297052 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:35.297105 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:35.300689 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.301673 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.301679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.301683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.301856 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.302368 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.302373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.302376 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.302729 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:35.302966 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:35.303557 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 14:34:35.303792 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:34:35.318262 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.276893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45166 (local address 127.6.46.252:37173)
0810 14:34:35.277049 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:35.277054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:35.290344 (+ 13290us) server_negotiation.cc:407] Connection header received
0810 14:34:35.290792 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:35.290798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:35.290876 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:35.290941 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:35.296204 (+ 5263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.297328 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.302401 (+ 5073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.303153 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.303250 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.307201 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:35.307210 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:35.307215 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:35.307253 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:35.314149 (+ 6896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.315126 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.315132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.315136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.315290 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.315767 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.315772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.315776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.316154 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:35.316408 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:35.317424 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0810 14:34:35.317675 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:34:35.318960 29081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.280055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48326 (local address 127.6.46.253:39717)
0810 14:34:35.280206 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:35.280211 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:35.293551 (+ 13340us) server_negotiation.cc:407] Connection header received
0810 14:34:35.293808 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:35.293813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:35.293887 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:35.293956 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:35.299426 (+ 5470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.300584 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.307150 (+ 6566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.307791 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.308165 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.311619 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:35.311626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:35.311631 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:35.311668 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:35.315064 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.315943 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.315948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.315952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.316126 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.316554 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.316559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.316562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.316915 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:35.317169 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:35.318195 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0810 14:34:35.318626 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 14:34:35.328342 29254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:34:35.328472 29254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:34:35.329278 29085 master_runner.cc:186] Error getting master registration for 127.6.46.253:39717: OK, instance_id { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" instance_seqno: 1786372474694739 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:34:35.340070 29085 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:34:35.411940 29073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } attempt: 5
W20260810 14:34:35.424027 29118 tablet.cc:2369] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 14:34:35 dist-test-slave-q9h9 krb5kdc[28817](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372474, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.253@KRBTEST.COM for kudu/127.6.46.252@KRBTEST.COM
I20260810 14:34:35.433342 29079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.412482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:37173 (local address 127.0.0.1:45182)
0810 14:34:35.412777 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:35.412797 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:35.412946 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:35.413293 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:35.413298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:35.413892 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:35.414135 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:35.414148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.415619 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.415625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:35.417070 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:35.417086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.418141 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.418147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:35.418362 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.423667 (+ 5305us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:35.423694 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:35.426578 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:35.430606 (+ 4028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:35.430612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:35.430616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:35.431151 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:35.431527 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:35.431531 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:35.431534 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:35.431710 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:35.432357 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:35.432362 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:35.432522 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 14:34:35.432695 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 14:34:35.432969 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260810 14:34:35.433549 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:35.412560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.6.46.252:37173)
0810 14:34:35.412772 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:35.412778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:35.412896 (+ 118us) server_negotiation.cc:407] Connection header received
0810 14:34:35.413072 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:35.413078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:35.413156 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:35.413254 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:35.414284 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.415476 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.417222 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:35.417974 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:35.418136 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:35.426775 (+ 8639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:35.426782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:35.426787 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:35.426825 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:35.430470 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.431269 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.431275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.431279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.431420 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:35.431822 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:35.431827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:35.431831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:35.432234 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:35.432381 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:35.432829 (+ 448us) server_negotiation.cc:299] Negotiation successful
0810 14:34:35.433066 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:34:35.439309 29073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569: Bootstrap starting.
I20260810 14:34:35.443467 29073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:35.444726 29073 log.cc:824] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:35.447846 29073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569: No bootstrap required, opened a new log
I20260810 14:34:35.458480 29073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } }
I20260810 14:34:35.458853 29073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:35.458966 29073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77701590d5bd4af08a782d9ef98fd569, State: Initialized, Role: FOLLOWER
I20260810 14:34:35.459587 29073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } }
I20260810 14:34:35.460513 29258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } } }
I20260810 14:34:35.460971 29258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:34:35.461726 29073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 14:34:35.473555 28874 tablet.cc:2369] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:34:35.479612 28970 master_runner.cc:263] No leader master found from master 77701590d5bd4af08a782d9ef98fd569
I20260810 14:34:35.479743 28970 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 14:34:35.482230 29279 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:34:35.482414 29279 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:34:35.527159 29234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:34:35.527401 29234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } }
I20260810 14:34:35.528824 29234 leader_election.cc:288] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b07b596943c848a8acbe0da9984ab33d (127.6.46.254:46023), 77701590d5bd4af08a782d9ef98fd569 (127.6.46.253:39717)
I20260810 14:34:35.530038 28908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b07b596943c848a8acbe0da9984ab33d" is_pre_election: true
I20260810 14:34:35.530632 28908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eefaec2049c475ab4f62c5b6c59fa0a in term 0.
I20260810 14:34:35.531260 29033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77701590d5bd4af08a782d9ef98fd569" is_pre_election: true
I20260810 14:34:35.531759 29033 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6eefaec2049c475ab4f62c5b6c59fa0a in term 0.
I20260810 14:34:35.532347 29114 leader_election.cc:302] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eefaec2049c475ab4f62c5b6c59fa0a, 77701590d5bd4af08a782d9ef98fd569; no voters:
I20260810 14:34:35.533185 29234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:34:35.533325 29234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:34:35.533424 29234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:35.536697 29234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } }
I20260810 14:34:35.537715 28908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b07b596943c848a8acbe0da9984ab33d"
I20260810 14:34:35.537936 28908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:35.541287 28908 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eefaec2049c475ab4f62c5b6c59fa0a in term 1.
I20260810 14:34:35.541760 29112 leader_election.cc:302] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6eefaec2049c475ab4f62c5b6c59fa0a, b07b596943c848a8acbe0da9984ab33d; no voters:
I20260810 14:34:35.545723 29033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77701590d5bd4af08a782d9ef98fd569"
I20260810 14:34:35.545943 29033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:35.546394 29284 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:34:35.549199 29033 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6eefaec2049c475ab4f62c5b6c59fa0a in term 1.
I20260810 14:34:35.549782 29234 leader_election.cc:288] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [CANDIDATE]: Term 1 election: Requested vote from peers b07b596943c848a8acbe0da9984ab33d (127.6.46.254:46023), 77701590d5bd4af08a782d9ef98fd569 (127.6.46.253:39717)
I20260810 14:34:35.554286 29284 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [term 1 LEADER]: Becoming Leader. State: Replica: 6eefaec2049c475ab4f62c5b6c59fa0a, State: Running, Role: LEADER
I20260810 14:34:35.554913 29284 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } }
I20260810 14:34:35.571255 29234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eefaec2049c475ab4f62c5b6c59fa0a. Latest consensus state: current_term: 1 leader_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } } }
I20260810 14:34:35.571583 29234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:35.579136 29289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:34:35.581432 29289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:34:35.596450 29033 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [term 1 FOLLOWER]: Refusing update from remote peer 6eefaec2049c475ab4f62c5b6c59fa0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:34:35.597445 29234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:34:35.604630 28908 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [term 1 FOLLOWER]: Refusing update from remote peer 6eefaec2049c475ab4f62c5b6c59fa0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:34:35.605424 29234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:34:35.639999 29202 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eefaec2049c475ab4f62c5b6c59fa0a. Latest consensus state: current_term: 1 leader_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } } }
I20260810 14:34:35.640357 29202 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:34:35.651516 29290 mvcc.cc:205] Tried to move back new op lower bound from 7316981660031168512 to 7316981659878617088. Current Snapshot: MvccSnapshot[applied={T|T < 7316981660031168512}]
I20260810 14:34:35.655063 29292 mvcc.cc:205] Tried to move back new op lower bound from 7316981660031168512 to 7316981659878617088. Current Snapshot: MvccSnapshot[applied={T|T < 7316981660031168512}]
I20260810 14:34:35.662122 29258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6eefaec2049c475ab4f62c5b6c59fa0a. Latest consensus state: current_term: 1 leader_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } } }
I20260810 14:34:35.662467 29258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:34:35.662673 29287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } } }
I20260810 14:34:35.663046 29287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:35.664250 29202 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } } }
I20260810 14:34:35.664559 29202 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:34:35.667515 29284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } } }
I20260810 14:34:35.667811 29284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:35.670226 29258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b07b596943c848a8acbe0da9984ab33d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46023 } } peers { permanent_uuid: "77701590d5bd4af08a782d9ef98fd569" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 39717 } } peers { permanent_uuid: "6eefaec2049c475ab4f62c5b6c59fa0a" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 37173 } } }
I20260810 14:34:35.670459 29258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:34:35.671953 29289 catalog_manager.cc:1347] Generated new cluster ID: 51928144114346428b0db36cf853606f
I20260810 14:34:35.672063 29289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:34:35.715116 29289 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:34:35.716876 29289 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:34:35.743981 29289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6eefaec2049c475ab4f62c5b6c59fa0a: Generated new TSK 0
I20260810 14:34:35.744984 29289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:34:36.284739 29216 catalog_manager.cc:1259] Loaded cluster ID: 51928144114346428b0db36cf853606f
I20260810 14:34:36.284858 29216 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d: loading cluster ID for follower catalog manager: success
I20260810 14:34:36.287552 29216 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d: acquiring CA information for follower catalog manager: success
I20260810 14:34:36.289392 29216 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b07b596943c848a8acbe0da9984ab33d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:34:36.484329 29279 catalog_manager.cc:1259] Loaded cluster ID: 51928144114346428b0db36cf853606f
I20260810 14:34:36.484427 29279 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569: loading cluster ID for follower catalog manager: success
I20260810 14:34:36.486879 29279 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569: acquiring CA information for follower catalog manager: success
I20260810 14:34:36.488552 29279 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 77701590d5bd4af08a782d9ef98fd569: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:34:36.863108 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 28837
I20260810 14:34:36.864382 28946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:36.969822 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 28837
I20260810 14:34:36.998474 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 28970
I20260810 14:34:36.999516 29062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:37.101498 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 28970
I20260810 14:34:37.125397 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 29085
I20260810 14:34:37.126457 29187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:37.230592 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 29085
2026-08-10T14:34:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4878 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:34:37 dist-test-slave-q9h9 krb5kdc[29318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:34:37 dist-test-slave-q9h9 krb5kdc[29318](info): set up 2 sockets
Aug 10 14:34:37 dist-test-slave-q9h9 krb5kdc[29318](info): commencing operation
krb5kdc: starting...
W20260810 14:34:38.459355 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.183s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:34:38 dist-test-slave-q9h9 krb5kdc[29318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:34:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:34:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:34:38.684901 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:46159
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:46159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:34:39.000253 29337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:39.000521 29337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:39.000653 29337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:39.007584 29337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:34:39.007687 29337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:39.007761 29337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:39.007836 29337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:34:39.007895 29337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:34:39.017333 29337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:46159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:46159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:39.018496 29337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:39.019703 29337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:39.030434 29345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:39.030524 29349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:39.030475 29346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:39.031376 29337 server_base.cc:1034] running on GCE node
I20260810 14:34:39.032071 29337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:39.033272 29337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:39.034467 29337 hybrid_clock.cc:648] HybridClock initialized: now 1786372479034416 us; error 38 us; skew 500 ppm
I20260810 14:34:39.034797 29337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:39 dist-test-slave-q9h9 krb5kdc[29318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372479, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:39.037657 29337 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:34:39.038970 29337 webserver.cc:460] Webserver started at http://127.6.46.254:46559/ using document root <none> and password file <none>
I20260810 14:34:39.039623 29337 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:39.039721 29337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:39.039966 29337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:34:39.042742 29337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "70ee400624be47c8b80e0847399878b9"
format_stamp: "Formatted at 2026-08-10 14:34:39 on dist-test-slave-q9h9"
server_key: "534e9315fbbd3f267dbcacb065fe4c19"
server_key_iv: "e8d47724c5d616639220da5335f22cf7"
server_key_version: "encryptionkey@0"
I20260810 14:34:39.043382 29337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "70ee400624be47c8b80e0847399878b9"
format_stamp: "Formatted at 2026-08-10 14:34:39 on dist-test-slave-q9h9"
server_key: "534e9315fbbd3f267dbcacb065fe4c19"
server_key_iv: "e8d47724c5d616639220da5335f22cf7"
server_key_version: "encryptionkey@0"
I20260810 14:34:39.048132 29337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:34:39.051311 29360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:39.052484 29337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:34:39.052634 29337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "70ee400624be47c8b80e0847399878b9"
format_stamp: "Formatted at 2026-08-10 14:34:39 on dist-test-slave-q9h9"
server_key: "534e9315fbbd3f267dbcacb065fe4c19"
server_key_iv: "e8d47724c5d616639220da5335f22cf7"
server_key_version: "encryptionkey@0"
I20260810 14:34:39.052783 29337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:39.074910 29337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:39.078771 29337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:39.078979 29337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:39.095847 29337 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:46159
I20260810 14:34:39.095892 29440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:46159 every 8 connection(s)
I20260810 14:34:39.097048 29337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:34:39.100454 29441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:39.103258 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 29337
I20260810 14:34:39.103492 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:34:39.104423 6331 external_mini_cluster.cc:1428] Setting key 7964b93fd197150c5796869a4fd46633
I20260810 14:34:39.112317 29441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9: Bootstrap starting.
I20260810 14:34:39.115882 29441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:39.117096 29441 log.cc:824] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:34:39 dist-test-slave-q9h9 krb5kdc[29318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:39.119798 29441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9: No bootstrap required, opened a new log
I20260810 14:34:39.126361 29446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:39.106241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33516 (local address 127.6.46.254:46159)
0810 14:34:39.106901 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:39.106911 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:39.106949 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:34:39.108582 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:39.108603 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:39.108956 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:39.109263 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:39.111007 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:39.112261 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:39.114116 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:39.114754 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:39.114879 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:39.119499 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:39.119506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:39.119519 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:39.119556 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:39.122523 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:39.123193 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:39.123199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:39.123204 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:39.123394 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:39.123760 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:39.123764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:39.123769 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:39.124261 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:39.124550 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:39.125020 (+ 470us) server_negotiation.cc:299] Negotiation successful
0810 14:34:39.125381 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":289,"threads_started":1}
I20260810 14:34:39.127382 29441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ee400624be47c8b80e0847399878b9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46159 } }
I20260810 14:34:39.127714 29441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:39.127813 29441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70ee400624be47c8b80e0847399878b9, State: Initialized, Role: FOLLOWER
I20260810 14:34:39.128315 29441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ee400624be47c8b80e0847399878b9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46159 } }
I20260810 14:34:39.128554 29441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:34:39.128701 29441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:34:39.128875 29441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:39.129902 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:34:39.131690 29441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ee400624be47c8b80e0847399878b9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46159 } }
I20260810 14:34:39.132213 29441 leader_election.cc:302] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70ee400624be47c8b80e0847399878b9; no voters:
I20260810 14:34:39.132827 29441 leader_election.cc:288] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:34:39.132917 29450 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:34:39.134773 29441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:34:39.138406 29450 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [term 1 LEADER]: Becoming Leader. State: Replica: 70ee400624be47c8b80e0847399878b9, State: Running, Role: LEADER
I20260810 14:34:39.138998 29450 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ee400624be47c8b80e0847399878b9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46159 } }
I20260810 14:34:39.141954 29452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70ee400624be47c8b80e0847399878b9. Latest consensus state: current_term: 1 leader_uuid: "70ee400624be47c8b80e0847399878b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ee400624be47c8b80e0847399878b9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46159 } } }
I20260810 14:34:39.142096 29451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70ee400624be47c8b80e0847399878b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70ee400624be47c8b80e0847399878b9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 46159 } } }
I20260810 14:34:39.142859 29452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:39.142935 29451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9 [sys.catalog]: This master's current role is: LEADER
W20260810 14:34:39.146643 29474 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:34:39.146742 29474 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:34:39.146873 29476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:34:39.148963 29476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:34:39.157105 29476 catalog_manager.cc:1347] Generated new cluster ID: 860739ea991c4c2f9aa49193f41a0383
I20260810 14:34:39.157207 29476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:34:39.231511 29476 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:34:39.233316 29476 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:34:39.242250 29476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70ee400624be47c8b80e0847399878b9: Generated new TSK 0
I20260810 14:34:39.243192 29476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:34:39.293412 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 29337
I20260810 14:34:39.294517 29436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:39.393237 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 29337
2026-08-10T14:34:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2161 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:34:39 dist-test-slave-q9h9 krb5kdc[29487](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:34:39 dist-test-slave-q9h9 krb5kdc[29487](info): set up 2 sockets
Aug 10 14:34:39 dist-test-slave-q9h9 krb5kdc[29487](info): commencing operation
krb5kdc: starting...
W20260810 14:34:40.615908 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:34:40 dist-test-slave-q9h9 krb5kdc[29487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:34:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:34:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:34:40.848603 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36421
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:36421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:34:41.169474 29514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:41.169741 29514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:41.169886 29514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:41.177094 29514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:34:41.177197 29514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:41.177286 29514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:41.177376 29514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:34:41.177443 29514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:34:41.187203 29514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:36421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:41.188428 29514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:41.189699 29514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:41.200897 29525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:41.200958 29524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:41.200989 29530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:41.201879 29514 server_base.cc:1034] running on GCE node
I20260810 14:34:41.202592 29514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:41.203804 29514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:41.205000 29514 hybrid_clock.cc:648] HybridClock initialized: now 1786372481204956 us; error 32 us; skew 500 ppm
I20260810 14:34:41.205284 29514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:41 dist-test-slave-q9h9 krb5kdc[29487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372481, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:41.208267 29514 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:34:41.209630 29514 webserver.cc:460] Webserver started at http://127.6.46.254:37119/ using document root <none> and password file <none>
I20260810 14:34:41.210271 29514 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:41.210384 29514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:41.210664 29514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:34:41.213411 29514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "5ba0ce8dcb514d43a6e262bade39824b"
format_stamp: "Formatted at 2026-08-10 14:34:41 on dist-test-slave-q9h9"
server_key: "8592f20b3b8694f5bdace1191b1a95c8"
server_key_iv: "57ce9bc0010938ea1c4efb6c1e45841e"
server_key_version: "encryptionkey@0"
I20260810 14:34:41.214164 29514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "5ba0ce8dcb514d43a6e262bade39824b"
format_stamp: "Formatted at 2026-08-10 14:34:41 on dist-test-slave-q9h9"
server_key: "8592f20b3b8694f5bdace1191b1a95c8"
server_key_iv: "57ce9bc0010938ea1c4efb6c1e45841e"
server_key_version: "encryptionkey@0"
I20260810 14:34:41.218890 29514 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:34:41.222082 29540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:41.223241 29514 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:34:41.223383 29514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "5ba0ce8dcb514d43a6e262bade39824b"
format_stamp: "Formatted at 2026-08-10 14:34:41 on dist-test-slave-q9h9"
server_key: "8592f20b3b8694f5bdace1191b1a95c8"
server_key_iv: "57ce9bc0010938ea1c4efb6c1e45841e"
server_key_version: "encryptionkey@0"
I20260810 14:34:41.223551 29514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:41.248011 29514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:41.251984 29514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:41.252270 29514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:41.271008 29514 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:36421
I20260810 14:34:41.271041 29627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:36421 every 8 connection(s)
I20260810 14:34:41.272208 29514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:34:41.275897 29630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:41.278261 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 29514
I20260810 14:34:41.278456 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:34:41.279091 6331 external_mini_cluster.cc:1428] Setting key afb8d82111acbedf9786cb333130bfe2
I20260810 14:34:41.287132 29630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Bootstrap starting.
I20260810 14:34:41.290658 29630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:41.291875 29630 log.cc:824] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:34:41 dist-test-slave-q9h9 krb5kdc[29487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:41.294512 29630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: No bootstrap required, opened a new log
I20260810 14:34:41.300868 29637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:41.280976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52714 (local address 127.6.46.254:36421)
0810 14:34:41.281724 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:41.281735 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:41.281774 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:34:41.283363 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:41.283385 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:41.283753 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:41.284046 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:41.285749 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:41.287046 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:41.288777 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:41.289399 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:41.289516 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:41.294121 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:41.294129 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:41.294143 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:41.294177 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:41.297073 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:41.297726 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:41.297732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:41.297737 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:41.297892 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:41.298314 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:41.298318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:41.298320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:41.298830 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:41.299121 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:41.299568 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 14:34:41.299956 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":341,"threads_started":1}
I20260810 14:34:41.301934 29630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } }
I20260810 14:34:41.302284 29630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:41.302381 29630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba0ce8dcb514d43a6e262bade39824b, State: Initialized, Role: FOLLOWER
I20260810 14:34:41.302888 29630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } }
I20260810 14:34:41.303089 29630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:34:41.303232 29630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:34:41.303416 29630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:41.305544 29630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } }
I20260810 14:34:41.305968 29630 leader_election.cc:302] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ba0ce8dcb514d43a6e262bade39824b; no voters:
I20260810 14:34:41.306530 29630 leader_election.cc:288] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:34:41.306643 29640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:34:41.307243 29640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 1 LEADER]: Becoming Leader. State: Replica: 5ba0ce8dcb514d43a6e262bade39824b, State: Running, Role: LEADER
I20260810 14:34:41.307772 29640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } }
I20260810 14:34:41.308301 29630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:34:41.310663 29642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ba0ce8dcb514d43a6e262bade39824b. Latest consensus state: current_term: 1 leader_uuid: "5ba0ce8dcb514d43a6e262bade39824b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } } }
I20260810 14:34:41.311005 29642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:41.310632 29641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ba0ce8dcb514d43a6e262bade39824b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } } }
I20260810 14:34:41.311398 29641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:41.312407 29651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:34:41.316561 29651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:34:41.324051 29651 catalog_manager.cc:1347] Generated new cluster ID: 1d2671a6a66142a99aa77f7ff4e3e09e
I20260810 14:34:41.324158 29651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:34:41.333305 29651 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:34:41.334969 29651 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:34:41.341588 29651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Generated new TSK 0
I20260810 14:34:41.342450 29651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:34:41.401412 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36421
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:34:41.721300 29675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:41.721573 29675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:41.721676 29675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:41.728905 29675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:41.729015 29675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:41.729128 29675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:34:41.738838 29675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:41.740072 29675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:41.741333 29675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:41.753793 29689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:41.753821 29685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:41.753998 29684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:41.754631 29675 server_base.cc:1034] running on GCE node
I20260810 14:34:41.755189 29675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:41.756129 29675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:41.757318 29675 hybrid_clock.cc:648] HybridClock initialized: now 1786372481757270 us; error 39 us; skew 500 ppm
I20260810 14:34:41.757601 29675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:41 dist-test-slave-q9h9 krb5kdc[29487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372481, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:41.760663 29675 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:34:41.762012 29675 webserver.cc:460] Webserver started at http://127.6.46.193:45147/ using document root <none> and password file <none>
I20260810 14:34:41.762706 29675 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:41.762822 29675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:41.763103 29675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:34:41.765889 29675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "0439ed7a9a8d4b638990bc35aab79b5d"
format_stamp: "Formatted at 2026-08-10 14:34:41 on dist-test-slave-q9h9"
server_key: "527d8a2d364685638ec345407cf7ddf2"
server_key_iv: "8fc9d5212a52c812af2dcad56db5ce65"
server_key_version: "encryptionkey@0"
I20260810 14:34:41.766657 29675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "0439ed7a9a8d4b638990bc35aab79b5d"
format_stamp: "Formatted at 2026-08-10 14:34:41 on dist-test-slave-q9h9"
server_key: "527d8a2d364685638ec345407cf7ddf2"
server_key_iv: "8fc9d5212a52c812af2dcad56db5ce65"
server_key_version: "encryptionkey@0"
I20260810 14:34:41.771427 29675 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:34:41.774657 29700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:41.775843 29675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260810 14:34:41.775991 29675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "0439ed7a9a8d4b638990bc35aab79b5d"
format_stamp: "Formatted at 2026-08-10 14:34:41 on dist-test-slave-q9h9"
server_key: "527d8a2d364685638ec345407cf7ddf2"
server_key_iv: "8fc9d5212a52c812af2dcad56db5ce65"
server_key_version: "encryptionkey@0"
I20260810 14:34:41.776156 29675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:41.792305 29675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:41.796344 29675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:41.796607 29675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:41.797608 29675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:41.799018 29675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:41.799093 29675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:41.799177 29675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:41.799221 29675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:41.829811 29675 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:39199
I20260810 14:34:41.829857 29905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:39199 every 8 connection(s)
I20260810 14:34:41.831063 29675 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:34:41.833122 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 29675
I20260810 14:34:41.833315 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:34:41.834103 6331 external_mini_cluster.cc:1428] Setting key 7857a0071c6caf49a4e96f6a56ddf7d8
Aug 10 14:34:41 dist-test-slave-q9h9 krb5kdc[29487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372481, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:41.851248 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:41.833540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:47481 (local address 127.6.46.254:36421)
0810 14:34:41.834006 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:41.834039 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:41.837184 (+ 3145us) server_negotiation.cc:407] Connection header received
0810 14:34:41.837244 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:41.837249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:41.837326 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:41.837428 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:41.838931 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:41.839896 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:41.841551 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:41.842119 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:41.842237 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:41.845501 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:41.845507 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:41.845512 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:41.845548 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:41.848147 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:41.848994 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:41.848999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:41.849001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:41.849155 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:41.849602 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:41.849605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:41.849608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:41.849876 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:41.849987 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:41.850803 (+ 816us) server_negotiation.cc:299] Negotiation successful
0810 14:34:41.850975 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":225,"threads_started":1}
I20260810 14:34:41.852607 29910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:41.834039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:47481)
0810 14:34:41.834841 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:41.834878 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:41.836843 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:41.837657 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:41.837664 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:41.838218 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:41.838762 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:41.838780 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:41.840021 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:41.840024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:41.841402 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:41.841416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:41.842334 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:41.842339 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:41.842545 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:41.843161 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:41.843184 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:41.845273 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:41.848279 (+ 3006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:41.848288 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:41.848303 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:41.848873 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:41.849260 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:41.849265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:41.849268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:41.849514 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:41.849991 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:41.849999 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:41.850424 (+ 425us) client_negotiation.cc:769] Sending connection context
0810 14:34:41.850725 (+ 301us) client_negotiation.cc:240] Negotiation successful
0810 14:34:41.851263 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":302,"threads_started":1}
I20260810 14:34:41.854171 29906 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36421
I20260810 14:34:41.854451 29906 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:41.855165 29906 heartbeater.cc:511] Master 127.6.46.254:36421 requested a full tablet report, sending...
I20260810 14:34:41.857291 29567 ts_manager.cc:194] Registered new tserver with Master: 0439ed7a9a8d4b638990bc35aab79b5d (127.6.46.193:39199)
I20260810 14:34:41.859280 29567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:47481
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:34:41.895375 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36421
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:34:42.210538 29914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:42.210786 29914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:42.210886 29914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:42.217811 29914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:42.217906 29914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:42.218052 29914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:34:42.227141 29914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:42.228291 29914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:42.229523 29914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:42.246156 29924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:42.246181 29930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:42.246258 29923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:42.246980 29914 server_base.cc:1034] running on GCE node
I20260810 14:34:42.247543 29914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:42.248420 29914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:42.249612 29914 hybrid_clock.cc:648] HybridClock initialized: now 1786372482249561 us; error 44 us; skew 500 ppm
I20260810 14:34:42.249903 29914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:42 dist-test-slave-q9h9 krb5kdc[29487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372482, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:42.252825 29914 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:34:42.254158 29914 webserver.cc:460] Webserver started at http://127.6.46.194:43071/ using document root <none> and password file <none>
I20260810 14:34:42.254808 29914 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:42.254923 29914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:42.255196 29914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:34:42.257923 29914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "5f190e4521e24479b6f81542daf7ed8f"
format_stamp: "Formatted at 2026-08-10 14:34:42 on dist-test-slave-q9h9"
server_key: "3a5c921c678055a371552c0f6cb96961"
server_key_iv: "ed4a923b73ff9bfc5db2319572d2560d"
server_key_version: "encryptionkey@0"
I20260810 14:34:42.258659 29914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "5f190e4521e24479b6f81542daf7ed8f"
format_stamp: "Formatted at 2026-08-10 14:34:42 on dist-test-slave-q9h9"
server_key: "3a5c921c678055a371552c0f6cb96961"
server_key_iv: "ed4a923b73ff9bfc5db2319572d2560d"
server_key_version: "encryptionkey@0"
I20260810 14:34:42.263517 29914 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:34:42.266669 29941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:42.267907 29914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:34:42.268054 29914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "5f190e4521e24479b6f81542daf7ed8f"
format_stamp: "Formatted at 2026-08-10 14:34:42 on dist-test-slave-q9h9"
server_key: "3a5c921c678055a371552c0f6cb96961"
server_key_iv: "ed4a923b73ff9bfc5db2319572d2560d"
server_key_version: "encryptionkey@0"
I20260810 14:34:42.268221 29914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:42.283211 29914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:42.286990 29914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:42.287242 29914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:42.288118 29914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:42.289355 29914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:42.289433 29914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:42.289559 29914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:42.289614 29914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:42.319617 29914 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:35761
I20260810 14:34:42.319669 30144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:35761 every 8 connection(s)
I20260810 14:34:42.320804 29914 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:34:42.328044 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 29914
I20260810 14:34:42.328222 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:34:42.328632 6331 external_mini_cluster.cc:1428] Setting key 1076b8364daa7f895b7f06254693434b
Aug 10 14:34:42 dist-test-slave-q9h9 krb5kdc[29487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372482, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:42.340668 29909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:42.323120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:37257 (local address 127.6.46.254:36421)
0810 14:34:42.323299 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:42.323304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:42.324394 (+ 1090us) server_negotiation.cc:407] Connection header received
0810 14:34:42.326164 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:42.326170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:42.326249 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:42.326336 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:42.327750 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.328571 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.330527 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.331082 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.331236 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:42.334612 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:42.334619 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:42.334622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:42.334660 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:42.337608 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:42.338395 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:42.338399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:42.338402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:42.338565 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:42.338985 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:42.338989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:42.338992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:42.339313 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:42.339445 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:42.340146 (+ 701us) server_negotiation.cc:299] Negotiation successful
0810 14:34:42.340364 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:34:42.341549 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:42.323545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:37257)
0810 14:34:42.324245 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:42.324281 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:42.325887 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:42.326587 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:42.326595 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:42.327067 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:42.327577 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:42.327594 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.328734 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.328739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:42.330393 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:42.330409 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.331280 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.331285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:42.331472 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:42.332094 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:42.332114 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:42.334390 (+ 2276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:42.337744 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:42.337752 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:42.337766 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:42.338267 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:42.338667 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:42.338671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:42.338674 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:42.338879 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:42.339417 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:42.339424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:42.339781 (+ 357us) client_negotiation.cc:769] Sending connection context
0810 14:34:42.340019 (+ 238us) client_negotiation.cc:240] Negotiation successful
0810 14:34:42.340442 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":279,"threads_started":1}
I20260810 14:34:42.343308 30145 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36421
I20260810 14:34:42.343616 30145 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:42.344283 30145 heartbeater.cc:511] Master 127.6.46.254:36421 requested a full tablet report, sending...
I20260810 14:34:42.345624 29567 ts_manager.cc:194] Registered new tserver with Master: 5f190e4521e24479b6f81542daf7ed8f (127.6.46.194:35761)
I20260810 14:34:42.346635 29567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:37257
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:34:42.386444 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36421
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:34:42.719348 30152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:42.719622 30152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:42.719731 30152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:42.726760 30152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:42.726866 30152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:42.727035 30152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:34:42.736632 30152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:42.737797 30152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:42.739142 30152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:42.752040 30159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:42.752063 30164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:42.751976 30160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:42.752743 30152 server_base.cc:1034] running on GCE node
I20260810 14:34:42.753374 30152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:42.754319 30152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:42.755532 30152 hybrid_clock.cc:648] HybridClock initialized: now 1786372482755483 us; error 41 us; skew 500 ppm
I20260810 14:34:42.755913 30152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:42 dist-test-slave-q9h9 krb5kdc[29487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372482, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:42.759050 30152 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:34:42.760496 30152 webserver.cc:460] Webserver started at http://127.6.46.195:45457/ using document root <none> and password file <none>
I20260810 14:34:42.761191 30152 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:42.761309 30152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:42.761596 30152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:34:42.764494 30152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "5c81f4b07e7d40128063aa51b3bee2fb"
format_stamp: "Formatted at 2026-08-10 14:34:42 on dist-test-slave-q9h9"
server_key: "90b840cdfcd8d008fcada18af56b7409"
server_key_iv: "6c95082d7fe58b0bc4c2e46a8c44b01a"
server_key_version: "encryptionkey@0"
I20260810 14:34:42.765231 30152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "5c81f4b07e7d40128063aa51b3bee2fb"
format_stamp: "Formatted at 2026-08-10 14:34:42 on dist-test-slave-q9h9"
server_key: "90b840cdfcd8d008fcada18af56b7409"
server_key_iv: "6c95082d7fe58b0bc4c2e46a8c44b01a"
server_key_version: "encryptionkey@0"
I20260810 14:34:42.770586 30152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 14:34:42.773979 30175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:42.775332 30152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:34:42.775473 30152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "5c81f4b07e7d40128063aa51b3bee2fb"
format_stamp: "Formatted at 2026-08-10 14:34:42 on dist-test-slave-q9h9"
server_key: "90b840cdfcd8d008fcada18af56b7409"
server_key_iv: "6c95082d7fe58b0bc4c2e46a8c44b01a"
server_key_version: "encryptionkey@0"
I20260810 14:34:42.775653 30152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:42.801367 30152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:42.805444 30152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:42.805709 30152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:42.806703 30152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:42.808008 30152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:42.808086 30152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:42.808211 30152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:42.808266 30152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:42.839332 30152 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:41189
I20260810 14:34:42.839426 30380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:41189 every 8 connection(s)
I20260810 14:34:42.840499 30152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:34:42.849638 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 30152
I20260810 14:34:42.849820 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:34:42.850366 6331 external_mini_cluster.cc:1428] Setting key ba926ae7d6f2fa22d6878ba0df415e23
Aug 10 14:34:42 dist-test-slave-q9h9 krb5kdc[29487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372482, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:42.859519 30384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:42.842983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:44887 (local address 127.6.46.254:36421)
0810 14:34:42.843469 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:42.843475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:42.844319 (+ 844us) server_negotiation.cc:407] Connection header received
0810 14:34:42.846075 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:42.846081 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:42.846153 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:42.846246 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:42.847655 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.848418 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.850219 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.850680 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.850784 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:42.854531 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:42.854536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:42.854540 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:42.854572 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:42.857066 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:42.857669 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:42.857673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:42.857675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:42.857787 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:42.858155 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:42.858158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:42.858161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:42.858424 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:42.858535 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:42.859086 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 14:34:42.859255 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":239,"threads_started":1}
I20260810 14:34:42.860374 30385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:42.843458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:44887)
0810 14:34:42.844163 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:42.844208 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:42.845804 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:42.846432 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:42.846440 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:42.846905 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:42.847503 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:42.847521 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.848532 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.848535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:42.850038 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:42.850055 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.850898 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.850904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:42.851114 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:42.851863 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:42.851893 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:42.854330 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:42.857174 (+ 2844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:42.857181 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:42.857191 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:42.857572 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:42.857860 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:42.857863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:42.857865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:42.858064 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:42.858518 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:42.858523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:42.858807 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 14:34:42.859014 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 14:34:42.859355 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":269,"threads_started":1}
I20260810 14:34:42.861713 30381 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36421
I20260810 14:34:42.861989 30381 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:42.862701 30381 heartbeater.cc:511] Master 127.6.46.254:36421 requested a full tablet report, sending...
I20260810 14:34:42.862787 29906 heartbeater.cc:503] Master 127.6.46.254:36421 was elected leader, sending a full tablet report...
I20260810 14:34:42.864343 29565 ts_manager.cc:194] Registered new tserver with Master: 5c81f4b07e7d40128063aa51b3bee2fb (127.6.46.195:41189)
I20260810 14:34:42.865374 29565 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:44887
I20260810 14:34:42.865718 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:34:42.881954 30384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:42.870194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.6.46.254:36421)
0810 14:34:42.870335 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:42.870339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:42.870808 (+ 469us) server_negotiation.cc:407] Connection header received
0810 14:34:42.870974 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:42.870978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:42.871034 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:42.871119 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:42.872535 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.873264 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.874592 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.874986 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.875091 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:42.877213 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:42.877218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:42.877221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:42.877251 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:42.879521 (+ 2270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:42.880135 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:42.880140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:42.880144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:42.880271 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:42.880612 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:42.880618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:42.880621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:42.880923 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:42.881150 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:42.881445 (+ 295us) server_negotiation.cc:299] Negotiation successful
0810 14:34:42.881652 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:34:42.885565 29565 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:34:42.892158 29565 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:34:42.915251 30404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:42.903289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:35761 (local address 127.0.0.1:49880)
0810 14:34:42.904314 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:42.904365 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:42.904536 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:42.907649 (+ 3113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:42.907660 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:42.907716 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:42.908204 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:42.908215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.910365 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.910371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:42.912610 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:42.912626 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.914373 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.914380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:42.914559 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:42.914601 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 14:34:42.914734 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 14:34:42.914910 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":2051,"threads_started":1}
I20260810 14:34:42.915241 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:42.903537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.6.46.194:35761)
0810 14:34:42.906987 (+ 3450us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:42.906996 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:42.907028 (+ 32us) server_negotiation.cc:407] Connection header received
0810 14:34:42.907124 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:42.907132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:42.907387 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:42.908414 (+ 1027us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:42.908688 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.910210 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.912767 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.914191 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.914318 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:42.914512 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:42.914789 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 14:34:42.914952 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3304,"thread_start_us":3212,"threads_started":1}
I20260810 14:34:42.916385 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:42.904178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.6.46.195:41189)
0810 14:34:42.906590 (+ 2412us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:42.906601 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:42.907969 (+ 1368us) server_negotiation.cc:407] Connection header received
0810 14:34:42.908414 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:42.908421 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:42.908637 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:42.908809 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:42.909828 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.911459 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.913773 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.915080 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.915216 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:42.915427 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:42.915935 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 14:34:42.916053 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2228,"thread_start_us":2046,"threads_started":1}
I20260810 14:34:42.916949 30407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:42.904046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39199 (local address 127.0.0.1:53176)
0810 14:34:42.909258 (+ 5212us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:42.909273 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:42.909402 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:42.909924 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:42.909928 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:42.909953 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:42.910310 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:42.910323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.911620 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.911625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:42.913947 (+ 2322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:42.913963 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.916204 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.916211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:42.916379 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:42.916410 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 14:34:42.916488 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:34:42.916604 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5077,"mutex_wait_us":3,"thread_start_us":3501,"threads_started":1}
I20260810 14:34:42.917524 30413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:42.904708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.6.46.193:39199)
0810 14:34:42.907389 (+ 2681us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:42.907397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:42.909376 (+ 1979us) server_negotiation.cc:407] Connection header received
0810 14:34:42.909551 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:42.909556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:42.909748 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:42.909933 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:42.910469 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.911433 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.914118 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.915528 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.915672 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:42.915887 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:42.917167 (+ 1280us) server_negotiation.cc:299] Negotiation successful
0810 14:34:42.917303 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2525,"thread_start_us":2384,"threads_started":1}
I20260810 14:34:42.919059 30029 tablet_service.cc:1467] Processing CreateTablet for tablet b260984274ad41a1afd5f7ea9339de72 (DEFAULT_TABLE table=test-table [id=d4e6a158ed62482d8d3a385f9dbc9b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:34:42.920225 30029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b260984274ad41a1afd5f7ea9339de72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:42.920205 29780 tablet_service.cc:1467] Processing CreateTablet for tablet b260984274ad41a1afd5f7ea9339de72 (DEFAULT_TABLE table=test-table [id=d4e6a158ed62482d8d3a385f9dbc9b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:34:42.920878 30409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:42.903557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:41189 (local address 127.0.0.1:47598)
0810 14:34:42.907851 (+ 4294us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:42.907869 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:42.908190 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:42.909153 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:42.909161 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:42.909187 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:42.909585 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:42.909598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.911635 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.911641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:42.913623 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:42.913636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:42.915561 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:42.915568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:42.915726 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:42.915758 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 14:34:42.920353 (+ 4595us) client_negotiation.cc:240] Negotiation successful
0810 14:34:42.920484 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4170,"mutex_wait_us":51,"thread_start_us":5430,"threads_started":1}
I20260810 14:34:42.921627 29780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b260984274ad41a1afd5f7ea9339de72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:42.924256 30263 tablet_service.cc:1467] Processing CreateTablet for tablet b260984274ad41a1afd5f7ea9339de72 (DEFAULT_TABLE table=test-table [id=d4e6a158ed62482d8d3a385f9dbc9b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:34:42.925575 30263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b260984274ad41a1afd5f7ea9339de72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:42.932073 30414 tablet_bootstrap.cc:492] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d: Bootstrap starting.
I20260810 14:34:42.935401 30414 tablet_bootstrap.cc:654] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:42.936373 30414 log.cc:824] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:42.937252 30415 tablet_bootstrap.cc:492] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f: Bootstrap starting.
I20260810 14:34:42.939448 30418 tablet_bootstrap.cc:492] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb: Bootstrap starting.
I20260810 14:34:42.940572 30414 tablet_bootstrap.cc:492] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d: No bootstrap required, opened a new log
I20260810 14:34:42.940611 30415 tablet_bootstrap.cc:654] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:42.940855 30414 ts_tablet_manager.cc:1397] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260810 14:34:42.941845 30415 log.cc:824] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:42.942241 30418 tablet_bootstrap.cc:654] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:42.943552 30418 log.cc:824] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:42.945781 30415 tablet_bootstrap.cc:492] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f: No bootstrap required, opened a new log
I20260810 14:34:42.946080 30415 ts_tablet_manager.cc:1397] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 14:34:42.948704 30418 tablet_bootstrap.cc:492] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb: No bootstrap required, opened a new log
I20260810 14:34:42.948918 30418 ts_tablet_manager.cc:1397] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260810 14:34:42.951122 30414 raft_consensus.cc:348] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } }
I20260810 14:34:42.951617 30414 raft_consensus.cc:374] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:42.951741 30414 raft_consensus.cc:729] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0439ed7a9a8d4b638990bc35aab79b5d, State: Initialized, Role: FOLLOWER
I20260810 14:34:42.952431 30414 consensus_queue.cc:260] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } }
I20260810 14:34:42.956241 30418 raft_consensus.cc:348] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } }
I20260810 14:34:42.956545 30418 raft_consensus.cc:374] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:42.956313 30415 raft_consensus.cc:348] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } }
I20260810 14:34:42.956683 30418 raft_consensus.cc:729] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c81f4b07e7d40128063aa51b3bee2fb, State: Initialized, Role: FOLLOWER
I20260810 14:34:42.956825 30415 raft_consensus.cc:374] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:42.956971 30415 raft_consensus.cc:729] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f190e4521e24479b6f81542daf7ed8f, State: Initialized, Role: FOLLOWER
I20260810 14:34:42.957424 30418 consensus_queue.cc:260] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } }
I20260810 14:34:42.957672 30415 consensus_queue.cc:260] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } }
I20260810 14:34:42.957937 30414 ts_tablet_manager.cc:1428] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
I20260810 14:34:42.958906 30145 heartbeater.cc:503] Master 127.6.46.254:36421 was elected leader, sending a full tablet report...
I20260810 14:34:42.959503 30415 ts_tablet_manager.cc:1428] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260810 14:34:42.960639 30381 heartbeater.cc:503] Master 127.6.46.254:36421 was elected leader, sending a full tablet report...
I20260810 14:34:42.961699 30418 ts_tablet_manager.cc:1428] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb: Time spent starting tablet: real 0.013s user 0.008s sys 0.005s
W20260810 14:34:43.072561 30146 tablet.cc:2369] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:34:43.083231 29907 tablet.cc:2369] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:34:43.091923 30382 tablet.cc:2369] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:34:43.202510 30422 raft_consensus.cc:482] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:34:43.202801 30422 raft_consensus.cc:504] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } }
I20260810 14:34:43.204425 30422 leader_election.cc:288] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f190e4521e24479b6f81542daf7ed8f (127.6.46.194:35761), 5c81f4b07e7d40128063aa51b3bee2fb (127.6.46.195:41189)
I20260810 14:34:43.211190 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:43.204896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:39473 (local address 127.6.46.194:35761)
0810 14:34:43.205139 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:43.205145 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:43.205805 (+ 660us) server_negotiation.cc:407] Connection header received
0810 14:34:43.205962 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:43.205968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:43.206076 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:43.206161 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:43.206780 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.207513 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.209405 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.210240 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.210362 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:43.210436 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:43.210811 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 14:34:43.210937 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 14:34:43.211195 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:43.204899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:41877 (local address 127.6.46.195:41189)
0810 14:34:43.205132 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:43.205139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:43.205940 (+ 801us) server_negotiation.cc:407] Connection header received
0810 14:34:43.206123 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:43.206128 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:43.206207 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:43.206305 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:43.206875 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.207614 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.209456 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.210263 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.210420 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:43.210491 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:43.210828 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 14:34:43.210950 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 14:34:43.211328 30430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:43.204875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:41189 (local address 127.6.46.193:41877)
0810 14:34:43.205827 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:43.205843 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:43.205976 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:43.206332 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:43.206335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:43.206368 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:43.206704 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:43.206716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.207745 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.207752 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:43.209250 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:43.209263 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.210523 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.210530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:43.210654 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:43.210701 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 14:34:43.210785 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 14:34:43.210903 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"mutex_wait_us":37,"thread_start_us":579,"threads_started":1}
I20260810 14:34:43.211398 30428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:43.204853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:35761 (local address 127.6.46.193:39473)
0810 14:34:43.205701 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:43.205716 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:43.205833 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:43.206184 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:43.206189 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:43.206222 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:43.206632 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:43.206643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.207651 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.207656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:43.209247 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:43.209260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.210483 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.210489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:43.210643 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:43.210684 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 14:34:43.210773 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 14:34:43.210892 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"mutex_wait_us":27,"thread_start_us":477,"threads_started":1}
I20260810 14:34:43.212451 30300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b260984274ad41a1afd5f7ea9339de72" candidate_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" is_pre_election: true
I20260810 14:34:43.212469 30066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b260984274ad41a1afd5f7ea9339de72" candidate_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f190e4521e24479b6f81542daf7ed8f" is_pre_election: true
I20260810 14:34:43.212990 30066 raft_consensus.cc:2454] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0439ed7a9a8d4b638990bc35aab79b5d in term 0.
I20260810 14:34:43.212990 30300 raft_consensus.cc:2454] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0439ed7a9a8d4b638990bc35aab79b5d in term 0.
I20260810 14:34:43.213647 29705 leader_election.cc:302] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0439ed7a9a8d4b638990bc35aab79b5d, 5f190e4521e24479b6f81542daf7ed8f; no voters:
I20260810 14:34:43.214003 30422 raft_consensus.cc:2781] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:34:43.214159 30422 raft_consensus.cc:482] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:34:43.214234 30422 raft_consensus.cc:3037] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:43.216305 30422 raft_consensus.cc:504] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } }
I20260810 14:34:43.216817 30422 leader_election.cc:288] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [CANDIDATE]: Term 1 election: Requested vote from peers 5f190e4521e24479b6f81542daf7ed8f (127.6.46.194:35761), 5c81f4b07e7d40128063aa51b3bee2fb (127.6.46.195:41189)
I20260810 14:34:43.217211 30066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b260984274ad41a1afd5f7ea9339de72" candidate_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f190e4521e24479b6f81542daf7ed8f"
I20260810 14:34:43.217237 30300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b260984274ad41a1afd5f7ea9339de72" candidate_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c81f4b07e7d40128063aa51b3bee2fb"
I20260810 14:34:43.217370 30066 raft_consensus.cc:3037] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:43.217401 30300 raft_consensus.cc:3037] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:43.219507 30300 raft_consensus.cc:2454] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0439ed7a9a8d4b638990bc35aab79b5d in term 1.
I20260810 14:34:43.219529 30066 raft_consensus.cc:2454] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0439ed7a9a8d4b638990bc35aab79b5d in term 1.
I20260810 14:34:43.219875 29705 leader_election.cc:302] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0439ed7a9a8d4b638990bc35aab79b5d, 5c81f4b07e7d40128063aa51b3bee2fb, 5f190e4521e24479b6f81542daf7ed8f; no voters:
I20260810 14:34:43.220155 30422 raft_consensus.cc:2781] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:34:43.220542 30422 raft_consensus.cc:686] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [term 1 LEADER]: Becoming Leader. State: Replica: 0439ed7a9a8d4b638990bc35aab79b5d, State: Running, Role: LEADER
I20260810 14:34:43.220966 30422 consensus_queue.cc:237] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } }
I20260810 14:34:43.225469 29565 catalog_manager.cc:5495] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0439ed7a9a8d4b638990bc35aab79b5d (127.6.46.193). New cstate: current_term: 1 leader_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } health_report { overall_health: HEALTHY } } }
I20260810 14:34:43.277360 30413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:43.271447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53188 (local address 127.6.46.193:39199)
0810 14:34:43.271617 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:43.271623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:43.271646 (+ 23us) server_negotiation.cc:407] Connection header received
0810 14:34:43.271871 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:43.271875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:43.271970 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:43.272107 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:34:43.272537 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.273254 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.275357 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.275749 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.275840 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:43.276390 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:34:43.276868 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:34:43.276969 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:43.277090 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 14:34:43.277153 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:34:43.286197 30066 raft_consensus.cc:1261] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f [term 1 FOLLOWER]: Refusing update from remote peer 0439ed7a9a8d4b638990bc35aab79b5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:34:43.286233 30300 raft_consensus.cc:1261] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb [term 1 FOLLOWER]: Refusing update from remote peer 0439ed7a9a8d4b638990bc35aab79b5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:34:43.287448 30431 consensus_queue.cc:1035] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:34:43.287830 30422 consensus_queue.cc:1035] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:34:43.302580 30437 mvcc.cc:205] Tried to move back new op lower bound from 7316981691529707520 to 7316981691276505088. Current Snapshot: MvccSnapshot[applied={T|T < 7316981691529707520}]
I20260810 14:34:43.315032 29565 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:34:43.315248 29565 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:34:43.318464 29565 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Sending DeleteTablet for 3 replicas of tablet b260984274ad41a1afd5f7ea9339de72
I20260810 14:34:43.319473 30263 tablet_service.cc:1514] Processing DeleteTablet for tablet b260984274ad41a1afd5f7ea9339de72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:34:43 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:47598
I20260810 14:34:43.319533 30029 tablet_service.cc:1514] Processing DeleteTablet for tablet b260984274ad41a1afd5f7ea9339de72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:34:43 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:49880
I20260810 14:34:43.322757 29780 tablet_service.cc:1514] Processing DeleteTablet for tablet b260984274ad41a1afd5f7ea9339de72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:34:43 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:53176
I20260810 14:34:43.323665 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 29675
I20260810 14:34:43.334918 30451 tablet_replica.cc:330] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb: stopping tablet replica
I20260810 14:34:43.335462 30451 raft_consensus.cc:2229] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:34:43.336028 30451 raft_consensus.cc:2258] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:34:43.339226 30452 tablet_replica.cc:330] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f: stopping tablet replica
I20260810 14:34:43.339725 30452 raft_consensus.cc:2229] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:34:43.340255 30452 raft_consensus.cc:2258] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:34:43.342486 30452 ts_tablet_manager.cc:1905] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:34:43.342486 30451 ts_tablet_manager.cc:1905] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:34:43.343449 30453 tablet_replica.cc:330] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d: stopping tablet replica
I20260810 14:34:43.343804 30453 raft_consensus.cc:2229] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:34:43.344410 30453 raft_consensus.cc:2258] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:34:43.349953 30452 ts_tablet_manager.cc:1918] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:34:43.350252 30452 log.cc:1196] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/b260984274ad41a1afd5f7ea9339de72
I20260810 14:34:43.350865 30452 ts_tablet_manager.cc:1939] T b260984274ad41a1afd5f7ea9339de72 P 5f190e4521e24479b6f81542daf7ed8f: Deleting consensus metadata
Aug 10 14:34:43 dist-test-slave-q9h9 krb5kdc[29487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:34:43.352593 30453 ts_tablet_manager.cc:1905] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:34:43.352804 29545 catalog_manager.cc:4841] TS 5f190e4521e24479b6f81542daf7ed8f (127.6.46.194:35761): tablet b260984274ad41a1afd5f7ea9339de72 (table test-table [id=d4e6a158ed62482d8d3a385f9dbc9b6f]) successfully deleted
I20260810 14:34:43.359299 30451 ts_tablet_manager.cc:1918] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:34:43.359464 30451 log.cc:1196] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/b260984274ad41a1afd5f7ea9339de72
I20260810 14:34:43.359968 30451 ts_tablet_manager.cc:1939] T b260984274ad41a1afd5f7ea9339de72 P 5c81f4b07e7d40128063aa51b3bee2fb: Deleting consensus metadata
I20260810 14:34:43.359951 30453 ts_tablet_manager.cc:1918] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:34:43.360136 30453 log.cc:1196] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/b260984274ad41a1afd5f7ea9339de72
I20260810 14:34:43.360657 30453 ts_tablet_manager.cc:1939] T b260984274ad41a1afd5f7ea9339de72 P 0439ed7a9a8d4b638990bc35aab79b5d: Deleting consensus metadata
I20260810 14:34:43.361411 30413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:43.324828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53204 (local address 127.6.46.193:39199)
0810 14:34:43.325003 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:43.325009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:43.330660 (+ 5651us) server_negotiation.cc:407] Connection header received
0810 14:34:43.331610 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:43.331616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:43.331698 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:43.331767 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:43.336480 (+ 4713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.337648 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.339929 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.340666 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.340771 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:43.353136 (+ 12365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:43.353151 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:43.353167 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:43.353209 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:43.357307 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:43.358262 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:43.358272 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:43.358276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:43.358501 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:43.359013 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:43.359019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:43.359022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:43.359607 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:43.359907 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:43.360666 (+ 759us) server_negotiation.cc:299] Negotiation successful
0810 14:34:43.360976 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:34:43.362149 29542 catalog_manager.cc:4841] TS 0439ed7a9a8d4b638990bc35aab79b5d (127.6.46.193:39199): tablet b260984274ad41a1afd5f7ea9339de72 (table test-table [id=d4e6a158ed62482d8d3a385f9dbc9b6f]) successfully deleted
I20260810 14:34:43.362416 29544 catalog_manager.cc:4841] TS 5c81f4b07e7d40128063aa51b3bee2fb (127.6.46.195:41189): tablet b260984274ad41a1afd5f7ea9339de72 (table test-table [id=d4e6a158ed62482d8d3a385f9dbc9b6f]) successfully deleted
I20260810 14:34:43.363101 29898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:43.460881 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 29675
I20260810 14:34:43.488957 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 29914
Aug 10 14:34:43 dist-test-slave-q9h9 krb5kdc[29487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:34:43.502600 30405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:43.489691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.6.46.194:35761)
0810 14:34:43.489883 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:43.489888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:43.489923 (+ 35us) server_negotiation.cc:407] Connection header received
0810 14:34:43.490133 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:43.490137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:43.490203 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:43.490277 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:43.491473 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.492155 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.493329 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.493723 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.493811 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:43.497791 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:43.497804 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:43.497818 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:43.497852 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:43.500325 (+ 2473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:43.500899 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:43.500905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:43.500908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:43.501031 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:43.501317 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:43.501320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:43.501322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:43.501661 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:43.501879 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:43.502118 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 14:34:43.502276 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:34:43.503203 30131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:43.589056 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 29914
I20260810 14:34:43.607427 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 30152
Aug 10 14:34:43 dist-test-slave-q9h9 krb5kdc[29487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:34:43.621937 30411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:43.608219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.6.46.195:41189)
0810 14:34:43.608396 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:43.608402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:43.608422 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:34:43.608605 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:43.608609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:43.608677 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:43.608760 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:43.610178 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.610880 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.612187 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:43.612598 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:43.612707 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:43.616905 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:43.616919 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:43.616936 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:43.616969 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:43.619363 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:43.619973 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:43.619981 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:43.619983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:43.620127 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:43.620467 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:43.620471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:43.620473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:43.620826 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:43.621097 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:43.621400 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 14:34:43.621621 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:34:43.622581 30375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:43.714092 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 30152
I20260810 14:34:43.732795 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 29514
I20260810 14:34:43.734038 29622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:43.842113 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 29514
I20260810 14:34:43.869426 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36421
--webserver_interface=127.6.46.254
--webserver_port=37119
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:36421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:34:44.198930 30467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:44.199219 30467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:44.199366 30467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:44.206370 30467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:34:44.206475 30467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:44.206558 30467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:44.206638 30467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:34:44.206698 30467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:34:44.216169 30467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:36421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=37119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:44.217465 30467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:44.218745 30467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:44.242210 30483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:44.242142 30488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:44.242179 30484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:44.242985 30467 server_base.cc:1034] running on GCE node
I20260810 14:34:44.243757 30467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:44.245236 30467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:44.246452 30467 hybrid_clock.cc:648] HybridClock initialized: now 1786372484246394 us; error 50 us; skew 500 ppm
I20260810 14:34:44.246847 30467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:44 dist-test-slave-q9h9 krb5kdc[29487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372484, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:44.250380 30467 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:34:44.251965 30467 webserver.cc:460] Webserver started at http://127.6.46.254:37119/ using document root <none> and password file <none>
I20260810 14:34:44.252692 30467 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:44.252828 30467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:44.258591 30467 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 14:34:44.261366 30507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:44.262672 30467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 14:34:44.262835 30467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "5ba0ce8dcb514d43a6e262bade39824b"
format_stamp: "Formatted at 2026-08-10 14:34:41 on dist-test-slave-q9h9"
server_key: "8592f20b3b8694f5bdace1191b1a95c8"
server_key_iv: "57ce9bc0010938ea1c4efb6c1e45841e"
server_key_version: "encryptionkey@0"
I20260810 14:34:44.263859 30467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:44.280694 30467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:44.284907 30467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:44.285162 30467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:44.301632 30467 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:36421
I20260810 14:34:44.301643 30596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:36421 every 8 connection(s)
I20260810 14:34:44.303041 30467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:34:44.304112 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 30467
I20260810 14:34:44.304905 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:39199
--local_ip_for_outbound_sockets=127.6.46.193
--tserver_master_addrs=127.6.46.254:36421
--webserver_port=45147
--webserver_interface=127.6.46.193
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:34:44.309901 30602 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:34:44.313118 30602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Bootstrap starting.
I20260810 14:34:44.332417 30602 log.cc:824] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:44.344615 30602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:34:44.345110 30602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Bootstrap complete.
I20260810 14:34:44.354070 30602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } }
I20260810 14:34:44.354912 30602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba0ce8dcb514d43a6e262bade39824b, State: Initialized, Role: FOLLOWER
I20260810 14:34:44.355548 30602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } }
I20260810 14:34:44.355778 30602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:34:44.355904 30602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:34:44.356062 30602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:34:44.359992 30602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } }
I20260810 14:34:44.360477 30602 leader_election.cc:302] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ba0ce8dcb514d43a6e262bade39824b; no voters:
I20260810 14:34:44.361029 30602 leader_election.cc:288] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 14:34:44.361212 30610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:34:44.362154 30610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 2 LEADER]: Becoming Leader. State: Replica: 5ba0ce8dcb514d43a6e262bade39824b, State: Running, Role: LEADER
I20260810 14:34:44.362792 30610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } }
I20260810 14:34:44.363061 30602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:34:44.365525 30612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ba0ce8dcb514d43a6e262bade39824b. Latest consensus state: current_term: 2 leader_uuid: "5ba0ce8dcb514d43a6e262bade39824b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } } }
I20260810 14:34:44.365888 30612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:44.366351 30611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5ba0ce8dcb514d43a6e262bade39824b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } } }
I20260810 14:34:44.366665 30611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:44.378759 30620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:34:44.386037 30620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:34:44.387392 30620 catalog_manager.cc:1259] Loaded cluster ID: 1d2671a6a66142a99aa77f7ff4e3e09e
I20260810 14:34:44.387487 30620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:34:44.390533 30620 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 14:34:44.407143 30620 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 14:34:44.415912 30620 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:34:44.417532 30620 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Loaded TSK: 0
I20260810 14:34:44.419062 30620 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:34:44.637703 30606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:44.637959 30606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:44.638095 30606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:44.645277 30606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:44.645401 30606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:44.645555 30606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:34:44.655508 30606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:39199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=45147
--tserver_master_addrs=127.6.46.254:36421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:44.656777 30606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:44.658118 30606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:44.671346 30653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:44.671386 30654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:44.674571 30657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:44.679289 30606 server_base.cc:1034] running on GCE node
I20260810 14:34:44.680043 30606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:44.681252 30606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:44.682502 30606 hybrid_clock.cc:648] HybridClock initialized: now 1786372484682441 us; error 45 us; skew 500 ppm
I20260810 14:34:44.682899 30606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:44 dist-test-slave-q9h9 krb5kdc[29487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372484, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:44.691059 30606 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:34:44.692780 30606 webserver.cc:460] Webserver started at http://127.6.46.193:45147/ using document root <none> and password file <none>
I20260810 14:34:44.693709 30606 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:44.693864 30606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:44.700719 30606 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:34:44.703770 30665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:44.705225 30606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:34:44.705389 30606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "0439ed7a9a8d4b638990bc35aab79b5d"
format_stamp: "Formatted at 2026-08-10 14:34:41 on dist-test-slave-q9h9"
server_key: "527d8a2d364685638ec345407cf7ddf2"
server_key_iv: "8fc9d5212a52c812af2dcad56db5ce65"
server_key_version: "encryptionkey@0"
I20260810 14:34:44.706714 30606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:44.747428 30606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:44.752274 30606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:44.752547 30606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:44.753643 30606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:44.755259 30606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:44.755348 30606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:44.755430 30606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:44.755483 30606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:44.785059 30606 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:39199
I20260810 14:34:44.785125 30862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:39199 every 8 connection(s)
I20260810 14:34:44.786487 30606 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:34:44.790918 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 30606
I20260810 14:34:44.791509 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:35761
--local_ip_for_outbound_sockets=127.6.46.194
--tserver_master_addrs=127.6.46.254:36421
--webserver_port=43071
--webserver_interface=127.6.46.194
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:34:44 dist-test-slave-q9h9 krb5kdc[29487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372484, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:44.809092 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:44.789208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:60623 (local address 127.6.46.254:36421)
0810 14:34:44.789726 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:44.789737 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:44.790431 (+ 694us) server_negotiation.cc:407] Connection header received
0810 14:34:44.792664 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:44.792685 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:44.793124 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:44.793514 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:44.794947 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:44.796636 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:44.798219 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:44.799108 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:44.799283 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:44.802492 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:44.802499 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:44.802512 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:44.802546 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:44.805186 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:44.805952 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:44.805958 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:44.805963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:44.806164 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:44.806596 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:44.806598 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:44.806600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:44.807019 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:44.807168 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:44.807935 (+ 767us) server_negotiation.cc:299] Negotiation successful
0810 14:34:44.808228 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":206,"threads_started":1}
I20260810 14:34:44.809451 30867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:44.789413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:60623)
0810 14:34:44.790261 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:44.790316 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:44.792079 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:44.793628 (+ 1549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:44.793636 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:44.794169 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:44.794703 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:44.794731 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:44.796837 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:44.796841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:44.798085 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:44.798098 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:44.799298 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:44.799302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:44.799487 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:44.800076 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:44.800098 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:44.802225 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:44.805328 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:44.805336 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:44.805349 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:44.805843 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:44.806274 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:44.806279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:44.806282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:44.806503 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:44.807158 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:44.807165 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:44.807509 (+ 344us) client_negotiation.cc:769] Sending connection context
0810 14:34:44.807772 (+ 263us) client_negotiation.cc:240] Negotiation successful
0810 14:34:44.808299 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":315,"threads_started":1}
I20260810 14:34:44.812451 30863 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36421
I20260810 14:34:44.812757 30863 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:44.813450 30863 heartbeater.cc:511] Master 127.6.46.254:36421 requested a full tablet report, sending...
I20260810 14:34:44.815891 30533 ts_manager.cc:194] Registered new tserver with Master: 0439ed7a9a8d4b638990bc35aab79b5d (127.6.46.193:39199)
I20260810 14:34:44.817952 30533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:60623
W20260810 14:34:45.116662 30871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:45.116915 30871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:45.117012 30871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:45.123842 30871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:45.123939 30871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:45.124042 30871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:34:45.133179 30871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:35761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=43071
--tserver_master_addrs=127.6.46.254:36421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:45.134375 30871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:45.135627 30871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:45.147827 30880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:45.147878 30881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:45.147907 30883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:45.148644 30871 server_base.cc:1034] running on GCE node
I20260810 14:34:45.149176 30871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:45.150066 30871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:45.151260 30871 hybrid_clock.cc:648] HybridClock initialized: now 1786372485151206 us; error 48 us; skew 500 ppm
I20260810 14:34:45.151546 30871 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:45 dist-test-slave-q9h9 krb5kdc[29487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372485, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:45.154448 30871 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:34:45.155727 30871 webserver.cc:460] Webserver started at http://127.6.46.194:43071/ using document root <none> and password file <none>
I20260810 14:34:45.156358 30871 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:45.156478 30871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:45.161512 30871 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260810 14:34:45.164073 30892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:45.165369 30871 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:34:45.165541 30871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "5f190e4521e24479b6f81542daf7ed8f"
format_stamp: "Formatted at 2026-08-10 14:34:42 on dist-test-slave-q9h9"
server_key: "3a5c921c678055a371552c0f6cb96961"
server_key_iv: "ed4a923b73ff9bfc5db2319572d2560d"
server_key_version: "encryptionkey@0"
I20260810 14:34:45.166548 30871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:45.177609 30871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:45.181147 30871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:45.181378 30871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:45.182300 30871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:45.183506 30871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:45.183607 30871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:45.183710 30871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:45.183782 30871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:45.211477 30871 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:35761
I20260810 14:34:45.211549 31106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:35761 every 8 connection(s)
I20260810 14:34:45.212847 30871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:34:45.219892 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 30871
I20260810 14:34:45.220543 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:41189
--local_ip_for_outbound_sockets=127.6.46.195
--tserver_master_addrs=127.6.46.254:36421
--webserver_port=45457
--webserver_interface=127.6.46.195
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:34:45 dist-test-slave-q9h9 krb5kdc[29487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372485, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:45.232545 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.215223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:40223 (local address 127.6.46.254:36421)
0810 14:34:45.215443 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:45.215448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:45.216478 (+ 1030us) server_negotiation.cc:407] Connection header received
0810 14:34:45.218272 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:45.218278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:45.218357 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:45.218444 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:45.220098 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.220862 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.222988 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.223405 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.223496 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.227196 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:45.227201 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:45.227204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:45.227236 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:45.229607 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:45.230339 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:45.230343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:45.230345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:45.230470 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:45.230879 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:45.230883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:45.230885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:45.231155 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:45.231289 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:45.231952 (+ 663us) server_negotiation.cc:299] Negotiation successful
0810 14:34:45.232199 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 14:34:45.233382 31114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.215615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:40223)
0810 14:34:45.216318 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:45.216354 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:45.217986 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:45.218717 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:45.218725 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:45.219194 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:45.219903 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.219929 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.221019 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.221025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.222861 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.222876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.223617 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.223624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.223873 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.224514 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:45.224540 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:45.226988 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:45.229742 (+ 2754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:45.229752 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:45.229767 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:45.230222 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:45.230573 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:45.230578 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:45.230582 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:45.230766 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:45.231284 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:45.231291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:45.231581 (+ 290us) client_negotiation.cc:769] Sending connection context
0810 14:34:45.231815 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 14:34:45.232293 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":275,"threads_started":1}
I20260810 14:34:45.234788 31109 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36421
I20260810 14:34:45.235055 31109 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:45.235697 31109 heartbeater.cc:511] Master 127.6.46.254:36421 requested a full tablet report, sending...
I20260810 14:34:45.237061 30533 ts_manager.cc:194] Registered new tserver with Master: 5f190e4521e24479b6f81542daf7ed8f (127.6.46.194:35761)
I20260810 14:34:45.238214 30533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:40223
W20260810 14:34:45.545457 31115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:45.545709 31115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:45.545818 31115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:45.552816 31115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:45.552915 31115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:45.553028 31115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:34:45.562289 31115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:41189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=45457
--tserver_master_addrs=127.6.46.254:36421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:45.563431 31115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:45.564713 31115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:45.577129 31125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:45.577206 31121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:45.577167 31122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:45.578213 31115 server_base.cc:1034] running on GCE node
I20260810 14:34:45.578737 31115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:45.579556 31115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:45.580752 31115 hybrid_clock.cc:648] HybridClock initialized: now 1786372485580717 us; error 34 us; skew 500 ppm
I20260810 14:34:45.581027 31115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:45 dist-test-slave-q9h9 krb5kdc[29487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372485, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:45.583912 31115 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:34:45.585276 31115 webserver.cc:460] Webserver started at http://127.6.46.195:45457/ using document root <none> and password file <none>
I20260810 14:34:45.585932 31115 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:45.586086 31115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:45.591032 31115 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 14:34:45.593588 31135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:45.594874 31115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:34:45.595023 31115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "5c81f4b07e7d40128063aa51b3bee2fb"
format_stamp: "Formatted at 2026-08-10 14:34:42 on dist-test-slave-q9h9"
server_key: "90b840cdfcd8d008fcada18af56b7409"
server_key_iv: "6c95082d7fe58b0bc4c2e46a8c44b01a"
server_key_version: "encryptionkey@0"
I20260810 14:34:45.595948 31115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:45.618698 31115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:45.622215 31115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:45.622447 31115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:45.623358 31115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:45.624643 31115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:45.624744 31115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:45.624845 31115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:45.624915 31115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:45.657514 31115 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:41189
I20260810 14:34:45.657631 31344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:41189 every 8 connection(s)
I20260810 14:34:45.658986 31115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:34:45.668001 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 31115
Aug 10 14:34:45 dist-test-slave-q9h9 krb5kdc[29487](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372485, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:34:45.679170 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.661477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:38083 (local address 127.6.46.254:36421)
0810 14:34:45.661658 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:45.661663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:45.662761 (+ 1098us) server_negotiation.cc:407] Connection header received
0810 14:34:45.664585 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:45.664591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:45.664672 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:45.664760 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:45.666175 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.666910 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.668535 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.669050 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.669193 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.673090 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:45.673094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:45.673097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:45.673133 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:45.676135 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:45.676919 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:45.676925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:45.676929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:45.677077 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:45.677475 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:45.677480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:45.677484 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:45.677826 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:45.677976 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:45.678720 (+ 744us) server_negotiation.cc:299] Negotiation successful
0810 14:34:45.678905 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:34:45.680290 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.661882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:38083)
0810 14:34:45.662594 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:45.662650 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:45.664298 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:45.664983 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:45.664991 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:45.665458 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:34:45.665984 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.666001 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.667069 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.667074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.668400 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.668413 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.669481 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.669486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.669675 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.670334 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:34:45.670364 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:34:45.672889 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:34:45.676265 (+ 3376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:45.676271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:45.676281 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:45.676792 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:45.677187 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:34:45.677191 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:34:45.677193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:34:45.677362 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:34:45.677929 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:34:45.677934 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:34:45.678319 (+ 385us) client_negotiation.cc:769] Sending connection context
0810 14:34:45.678565 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 14:34:45.679014 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":310,"threads_started":1}
I20260810 14:34:45.681769 31347 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36421
I20260810 14:34:45.682073 31347 heartbeater.cc:464] Registering TS with master...
I20260810 14:34:45.682914 31347 heartbeater.cc:511] Master 127.6.46.254:36421 requested a full tablet report, sending...
I20260810 14:34:45.684518 30533 ts_manager.cc:194] Registered new tserver with Master: 5c81f4b07e7d40128063aa51b3bee2fb (127.6.46.195:41189)
I20260810 14:34:45.684799 31352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.668841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.6.46.254:36421)
0810 14:34:45.669356 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:45.669361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:45.670537 (+ 1176us) server_negotiation.cc:407] Connection header received
0810 14:34:45.670718 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:45.670721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:45.670789 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:45.670871 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:45.672544 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.673533 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.675354 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.675799 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.675894 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.678842 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:45.678848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:45.678853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:45.678887 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:45.682154 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:45.682937 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:45.682940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:45.682942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:45.683062 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:45.683437 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:45.683439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:45.683441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:45.683674 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:45.683907 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:45.684198 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 14:34:45.684397 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":276,"threads_started":1}
I20260810 14:34:45.685607 30533 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:38083
I20260810 14:34:45.687058 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:34:45.692797 31352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.687889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.6.46.254:36421)
0810 14:34:45.688060 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:45.688066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:45.688421 (+ 355us) server_negotiation.cc:407] Connection header received
0810 14:34:45.688584 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:45.688588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:45.688708 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:45.688823 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:34:45.689153 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.689784 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.691072 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.691469 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.691562 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.691956 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:34:45.692414 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:34:45.692511 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:45.692547 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 14:34:45.692599 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:34:45.693563 30533 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:34:45.700443 30533 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:34:45.739394 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.711822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.6.46.195:41189)
0810 14:34:45.730195 (+ 18373us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:45.730205 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:45.730735 (+ 530us) server_negotiation.cc:407] Connection header received
0810 14:34:45.731027 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:45.731037 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:45.731311 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:45.731479 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:45.732255 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.733674 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.737220 (+ 3546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.738131 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.738244 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.738481 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:45.738878 (+ 397us) server_negotiation.cc:299] Negotiation successful
0810 14:34:45.739050 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18155,"thread_start_us":18001,"threads_started":1}
I20260810 14:34:45.739755 31374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.712290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49908 (local address 127.6.46.194:35761)
0810 14:34:45.730970 (+ 18680us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:45.730978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:45.731496 (+ 518us) server_negotiation.cc:407] Connection header received
0810 14:34:45.732277 (+ 781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:45.732283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:45.732463 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:45.732648 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:45.733319 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.734880 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.736740 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.738125 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.738260 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.738446 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:45.739317 (+ 871us) server_negotiation.cc:299] Negotiation successful
0810 14:34:45.739478 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18507,"thread_start_us":18370,"threads_started":1}
I20260810 14:34:45.739796 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.711283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53218 (local address 127.6.46.193:39199)
0810 14:34:45.731955 (+ 20672us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:45.731961 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:45.731980 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:34:45.732058 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:45.732062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:45.732243 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:45.732446 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:45.733051 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.734620 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.737007 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.738423 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.738591 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.738762 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:45.739376 (+ 614us) server_negotiation.cc:299] Negotiation successful
0810 14:34:45.739506 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20549,"thread_start_us":20424,"threads_started":1}
I20260810 14:34:45.740128 31362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.711204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:41189 (local address 127.0.0.1:47618)
0810 14:34:45.730620 (+ 19416us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:45.730640 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:45.730844 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:45.731682 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:45.731690 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:45.731733 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:45.732066 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.732076 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.733800 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.733804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.737086 (+ 3282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.737099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.738321 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.738328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.738503 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.738559 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 14:34:45.738706 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 14:34:45.738889 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19295,"mutex_wait_us":1,"thread_start_us":13737,"threads_started":1}
I20260810 14:34:45.740408 31364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.711097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39199 (local address 127.0.0.1:53218)
0810 14:34:45.725318 (+ 14221us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:45.725358 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:45.725516 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:45.732444 (+ 6928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:45.732449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:45.732472 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:45.732875 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.732888 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.734798 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.734807 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.736853 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.736868 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.738678 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.738685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.738887 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.738934 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 14:34:45.739018 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 14:34:45.739143 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13964,"mutex_wait_us":77,"thread_start_us":19255,"threads_started":1}
I20260810 14:34:45.740650 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.711619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:35761 (local address 127.0.0.1:49908)
0810 14:34:45.731371 (+ 19752us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:45.731387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:45.731502 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:45.732690 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:45.732694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:45.732725 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:45.733131 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.733143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.734985 (+ 1842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.734990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.736582 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.736595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.738628 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.738635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.738825 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.738944 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 14:34:45.739034 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 14:34:45.739153 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19647,"mutex_wait_us":70,"thread_start_us":19523,"threads_started":1}
I20260810 14:34:45.745471 30996 tablet_service.cc:1467] Processing CreateTablet for tablet 44171b190d8b4dae95f2699944bd6fd2 (DEFAULT_TABLE table=test-table [id=3b861e11fdd64947934975bee24d2b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:34:45.745471 30744 tablet_service.cc:1467] Processing CreateTablet for tablet 44171b190d8b4dae95f2699944bd6fd2 (DEFAULT_TABLE table=test-table [id=3b861e11fdd64947934975bee24d2b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:34:45.746969 30744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44171b190d8b4dae95f2699944bd6fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:45.746968 30996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44171b190d8b4dae95f2699944bd6fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:45.753423 31229 tablet_service.cc:1467] Processing CreateTablet for tablet 44171b190d8b4dae95f2699944bd6fd2 (DEFAULT_TABLE table=test-table [id=3b861e11fdd64947934975bee24d2b17]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:34:45.754705 31229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44171b190d8b4dae95f2699944bd6fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:34:45.768992 31381 tablet_bootstrap.cc:492] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb: Bootstrap starting.
I20260810 14:34:45.769506 31378 tablet_bootstrap.cc:492] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f: Bootstrap starting.
I20260810 14:34:45.772464 31380 tablet_bootstrap.cc:492] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d: Bootstrap starting.
I20260810 14:34:45.772485 31381 tablet_bootstrap.cc:654] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:45.773566 31381 log.cc:824] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:45.775818 31380 tablet_bootstrap.cc:654] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:45.776993 31378 tablet_bootstrap.cc:654] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f: Neither blocks nor log segments found. Creating new log.
I20260810 14:34:45.777138 31380 log.cc:824] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:45.778368 31378 log.cc:824] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:45.783268 31378 tablet_bootstrap.cc:492] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f: No bootstrap required, opened a new log
I20260810 14:34:45.783555 31378 ts_tablet_manager.cc:1397] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260810 14:34:45.784384 31381 tablet_bootstrap.cc:492] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb: No bootstrap required, opened a new log
I20260810 14:34:45.784741 31381 ts_tablet_manager.cc:1397] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.008s
I20260810 14:34:45.795217 31381 raft_consensus.cc:348] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } }
I20260810 14:34:45.795733 31381 raft_consensus.cc:374] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:45.795835 31381 raft_consensus.cc:729] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c81f4b07e7d40128063aa51b3bee2fb, State: Initialized, Role: FOLLOWER
I20260810 14:34:45.796078 31380 tablet_bootstrap.cc:492] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d: No bootstrap required, opened a new log
I20260810 14:34:45.795801 31378 raft_consensus.cc:348] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } }
I20260810 14:34:45.796200 31378 raft_consensus.cc:374] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:45.796296 31378 raft_consensus.cc:729] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f190e4521e24479b6f81542daf7ed8f, State: Initialized, Role: FOLLOWER
I20260810 14:34:45.796370 31380 ts_tablet_manager.cc:1397] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.012s
I20260810 14:34:45.796635 31381 consensus_queue.cc:260] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } }
I20260810 14:34:45.796983 31378 consensus_queue.cc:260] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } }
I20260810 14:34:45.797875 31347 heartbeater.cc:503] Master 127.6.46.254:36421 was elected leader, sending a full tablet report...
I20260810 14:34:45.798624 31381 ts_tablet_manager.cc:1428] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260810 14:34:45.815137 31380 raft_consensus.cc:348] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } }
I20260810 14:34:45.815613 31380 raft_consensus.cc:374] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:34:45.815707 31380 raft_consensus.cc:729] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0439ed7a9a8d4b638990bc35aab79b5d, State: Initialized, Role: FOLLOWER
I20260810 14:34:45.816368 31380 consensus_queue.cc:260] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } }
I20260810 14:34:45.817456 30863 heartbeater.cc:503] Master 127.6.46.254:36421 was elected leader, sending a full tablet report...
I20260810 14:34:45.817490 31109 heartbeater.cc:503] Master 127.6.46.254:36421 was elected leader, sending a full tablet report...
I20260810 14:34:45.818279 31378 ts_tablet_manager.cc:1428] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f: Time spent starting tablet: real 0.035s user 0.024s sys 0.009s
I20260810 14:34:45.819537 31380 ts_tablet_manager.cc:1428] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d: Time spent starting tablet: real 0.023s user 0.009s sys 0.006s
I20260810 14:34:45.830493 31390 raft_consensus.cc:482] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:34:45.830735 31390 raft_consensus.cc:504] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } }
I20260810 14:34:45.832140 31390 leader_election.cc:288] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5c81f4b07e7d40128063aa51b3bee2fb (127.6.46.195:41189), 5f190e4521e24479b6f81542daf7ed8f (127.6.46.194:35761)
I20260810 14:34:45.839073 31374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.833317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:53457 (local address 127.6.46.194:35761)
0810 14:34:45.833506 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:45.833512 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:45.833530 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:34:45.833591 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:45.833596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:45.833676 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:45.833745 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:45.834563 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.835449 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.837356 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.838178 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.838313 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.838386 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:45.838697 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 14:34:45.838816 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:34:45.839155 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.832583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:53851 (local address 127.6.46.195:41189)
0810 14:34:45.832755 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:45.832759 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:45.833547 (+ 788us) server_negotiation.cc:407] Connection header received
0810 14:34:45.833694 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:45.833697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:45.833765 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:45.833852 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:34:45.834596 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.835449 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.837388 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.838268 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.838411 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.838482 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:45.838786 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 14:34:45.838910 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:34:45.839339 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.832797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:41189 (local address 127.6.46.193:53851)
0810 14:34:45.833432 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:45.833450 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:45.833583 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:45.833884 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:45.833888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:45.833921 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:45.834380 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.834394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.835580 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.835584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.837207 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.837219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.838475 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.838481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.838645 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.838668 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:34:45.838745 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:34:45.838860 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":93,"thread_start_us":543,"threads_started":1}
I20260810 14:34:45.839601 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.832415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:35761 (local address 127.6.46.193:53457)
0810 14:34:45.832995 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:45.833015 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:34:45.833160 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:34:45.833785 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:34:45.833790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:34:45.833815 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:34:45.834391 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.834403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.835590 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.835594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.837210 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:34:45.837222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.838375 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.838381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:34:45.838539 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.838566 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 14:34:45.838650 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 14:34:45.838772 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":72,"spinlock_wait_cycles":23808,"thread_start_us":620,"threads_started":1}
I20260810 14:34:45.840224 31269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44171b190d8b4dae95f2699944bd6fd2" candidate_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" is_pre_election: true
I20260810 14:34:45.840386 31022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44171b190d8b4dae95f2699944bd6fd2" candidate_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f190e4521e24479b6f81542daf7ed8f" is_pre_election: true
I20260810 14:34:45.840646 31269 raft_consensus.cc:2454] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0439ed7a9a8d4b638990bc35aab79b5d in term 0.
I20260810 14:34:45.840837 31022 raft_consensus.cc:2454] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0439ed7a9a8d4b638990bc35aab79b5d in term 0.
I20260810 14:34:45.841274 30669 leader_election.cc:302] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0439ed7a9a8d4b638990bc35aab79b5d, 5c81f4b07e7d40128063aa51b3bee2fb; no voters:
I20260810 14:34:45.841661 31390 raft_consensus.cc:2781] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:34:45.841792 31390 raft_consensus.cc:482] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:34:45.841878 31390 raft_consensus.cc:3037] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:45.844309 31390 raft_consensus.cc:504] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } }
I20260810 14:34:45.844859 31390 leader_election.cc:288] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [CANDIDATE]: Term 1 election: Requested vote from peers 5c81f4b07e7d40128063aa51b3bee2fb (127.6.46.195:41189), 5f190e4521e24479b6f81542daf7ed8f (127.6.46.194:35761)
I20260810 14:34:45.845248 31269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44171b190d8b4dae95f2699944bd6fd2" candidate_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c81f4b07e7d40128063aa51b3bee2fb"
I20260810 14:34:45.845268 31022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44171b190d8b4dae95f2699944bd6fd2" candidate_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f190e4521e24479b6f81542daf7ed8f"
I20260810 14:34:45.845415 31022 raft_consensus.cc:3037] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:45.845418 31269 raft_consensus.cc:3037] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:34:45.847575 31022 raft_consensus.cc:2454] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0439ed7a9a8d4b638990bc35aab79b5d in term 1.
I20260810 14:34:45.847642 31269 raft_consensus.cc:2454] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0439ed7a9a8d4b638990bc35aab79b5d in term 1.
I20260810 14:34:45.847965 30670 leader_election.cc:302] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0439ed7a9a8d4b638990bc35aab79b5d, 5f190e4521e24479b6f81542daf7ed8f; no voters:
I20260810 14:34:45.848251 31390 raft_consensus.cc:2781] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:34:45.848634 31390 raft_consensus.cc:686] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [term 1 LEADER]: Becoming Leader. State: Replica: 0439ed7a9a8d4b638990bc35aab79b5d, State: Running, Role: LEADER
I20260810 14:34:45.849053 31390 consensus_queue.cc:237] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } } peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } }
I20260810 14:34:45.853399 30533 catalog_manager.cc:5495] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0439ed7a9a8d4b638990bc35aab79b5d (127.6.46.193). New cstate: current_term: 1 leader_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0439ed7a9a8d4b638990bc35aab79b5d" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 } health_report { overall_health: UNKNOWN } } }
I20260810 14:34:45.875660 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.870650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53234 (local address 127.6.46.193:39199)
0810 14:34:45.870843 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:45.870847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:45.870863 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:34:45.871034 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:45.871038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:45.871109 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:45.871222 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:34:45.871564 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.872257 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.873571 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.874075 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.874164 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.874670 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:34:45.875130 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:34:45.875243 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:45.875340 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 14:34:45.875402 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:34:45.883201 31269 raft_consensus.cc:1261] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb [term 1 FOLLOWER]: Refusing update from remote peer 0439ed7a9a8d4b638990bc35aab79b5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:34:45.883203 31022 raft_consensus.cc:1261] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f [term 1 FOLLOWER]: Refusing update from remote peer 0439ed7a9a8d4b638990bc35aab79b5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:34:45.884227 31395 consensus_queue.cc:1035] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c81f4b07e7d40128063aa51b3bee2fb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 41189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:34:45.884562 31390 consensus_queue.cc:1035] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f190e4521e24479b6f81542daf7ed8f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 35761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:34:45.894744 31397 mvcc.cc:205] Tried to move back new op lower bound from 7316981702168195072 to 7316981702041124864. Current Snapshot: MvccSnapshot[applied={T|T < 7316981702168195072}]
I20260810 14:34:45.909657 30533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:34:45.909875 30533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:34:45.913527 30533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Sending DeleteTablet for 3 replicas of tablet 44171b190d8b4dae95f2699944bd6fd2
I20260810 14:34:45.914608 30744 tablet_service.cc:1514] Processing DeleteTablet for tablet 44171b190d8b4dae95f2699944bd6fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:34:45 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:53218
I20260810 14:34:45.915081 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 30606
I20260810 14:34:45.915015 30996 tablet_service.cc:1514] Processing DeleteTablet for tablet 44171b190d8b4dae95f2699944bd6fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:34:45 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:49908
I20260810 14:34:45.915294 31229 tablet_service.cc:1514] Processing DeleteTablet for tablet 44171b190d8b4dae95f2699944bd6fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:34:45 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:47618
I20260810 14:34:45.916083 31412 tablet_replica.cc:330] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb: stopping tablet replica
I20260810 14:34:45.916476 31412 raft_consensus.cc:2229] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:34:45.916946 31412 raft_consensus.cc:2258] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:34:45.918538 31410 tablet_replica.cc:330] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f: stopping tablet replica
I20260810 14:34:45.918903 31410 raft_consensus.cc:2229] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:34:45.919333 31410 raft_consensus.cc:2258] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:34:45.920173 31412 ts_tablet_manager.cc:1905] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:34:45.923671 31410 ts_tablet_manager.cc:1905] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:34:45.925868 31412 ts_tablet_manager.cc:1918] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:34:45.927894 31412 log.cc:1196] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/44171b190d8b4dae95f2699944bd6fd2
I20260810 14:34:45.928474 31412 ts_tablet_manager.cc:1939] T 44171b190d8b4dae95f2699944bd6fd2 P 5c81f4b07e7d40128063aa51b3bee2fb: Deleting consensus metadata
I20260810 14:34:45.929286 31409 tablet_replica.cc:330] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d: stopping tablet replica
I20260810 14:34:45.929740 31409 raft_consensus.cc:2229] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:34:45.930460 31409 raft_consensus.cc:2258] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:34:45.931576 31410 ts_tablet_manager.cc:1918] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:34:45.931734 31410 log.cc:1196] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/44171b190d8b4dae95f2699944bd6fd2
I20260810 14:34:45.932291 31410 ts_tablet_manager.cc:1939] T 44171b190d8b4dae95f2699944bd6fd2 P 5f190e4521e24479b6f81542daf7ed8f: Deleting consensus metadata
I20260810 14:34:45.932565 30513 catalog_manager.cc:4841] TS 5c81f4b07e7d40128063aa51b3bee2fb (127.6.46.195:41189): tablet 44171b190d8b4dae95f2699944bd6fd2 (table test-table [id=3b861e11fdd64947934975bee24d2b17]) successfully deleted
I20260810 14:34:45.933938 30514 catalog_manager.cc:4841] TS 5f190e4521e24479b6f81542daf7ed8f (127.6.46.194:35761): tablet 44171b190d8b4dae95f2699944bd6fd2 (table test-table [id=3b861e11fdd64947934975bee24d2b17]) successfully deleted
I20260810 14:34:45.934329 31409 ts_tablet_manager.cc:1905] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:34:45.940418 31365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:45.916107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.6.46.193:39199)
0810 14:34:45.916270 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:45.916276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:45.916359 (+ 83us) server_negotiation.cc:407] Connection header received
0810 14:34:45.916534 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:45.916539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:45.916618 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:45.916734 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:45.918523 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.919755 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.927868 (+ 8113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:45.928499 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:45.928648 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:45.932929 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:45.932940 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:45.932956 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:45.932998 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:45.936917 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:45.937876 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:45.937883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:45.937887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:45.938095 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:45.938533 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:45.938538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:45.938541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:45.939007 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:45.939355 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:45.939702 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 14:34:45.939922 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:34:45.941047 31409 ts_tablet_manager.cc:1918] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:34:45.941175 30856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:45.941205 31409 log.cc:1196] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/44171b190d8b4dae95f2699944bd6fd2
I20260810 14:34:46.035982 31409 ts_tablet_manager.cc:1939] T 44171b190d8b4dae95f2699944bd6fd2 P 0439ed7a9a8d4b638990bc35aab79b5d: Deleting consensus metadata
I20260810 14:34:46.036149 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 30606
W20260810 14:34:46.062139 30509 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.46.193:39199 (error 108)
I20260810 14:34:46.062300 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 30871
I20260810 14:34:46.064049 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:46.063580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39199 (local address 127.0.0.1:53260)
0810 14:34:46.063770 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:46.063884 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260810 14:34:46.064297 30509 catalog_manager.cc:4584] TS 0439ed7a9a8d4b638990bc35aab79b5d (127.6.46.193:39199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 44171b190d8b4dae95f2699944bd6fd2: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39199: connect: Connection refused (error 111)
I20260810 14:34:46.075995 31374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:46.063074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49922 (local address 127.6.46.194:35761)
0810 14:34:46.063234 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:46.063238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:46.063256 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:34:46.063491 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:46.063497 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:46.063580 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:46.063685 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:46.065398 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:46.066347 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:46.067703 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:46.068217 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:46.068347 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:46.070924 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:46.070936 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:46.070950 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:46.070995 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:46.073634 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:46.074357 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:46.074363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:46.074366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:46.074495 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:46.074811 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:46.074814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:46.074816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:46.075166 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:46.075382 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:46.075584 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 14:34:46.075741 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:34:46.076663 31099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:46.118212 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:46.117678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39199 (local address 127.0.0.1:53262)
0810 14:34:46.117899 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:46.118003 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 14:34:46.119220 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:46.118846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39199 (local address 127.0.0.1:53276)
0810 14:34:46.118997 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:46.119087 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 14:34:46.162844 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 30871
I20260810 14:34:46.179947 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:46.179314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39199 (local address 127.0.0.1:53284)
0810 14:34:46.179613 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:46.179758 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 14:34:46.181072 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:46.180675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39199 (local address 127.0.0.1:53292)
0810 14:34:46.180855 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:46.180948 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 14:34:46.184846 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 31115
I20260810 14:34:46.197815 31372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:46.185532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47634 (local address 127.6.46.195:41189)
0810 14:34:46.185700 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 14:34:46.185706 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:34:46.185726 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:34:46.185939 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:34:46.185943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:34:46.186059 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:34:46.186144 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:34:46.187511 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:46.188209 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:46.189594 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:34:46.189981 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:34:46.190201 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:34:46.192667 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:34:46.192682 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:34:46.192697 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:34:46.192729 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:34:46.195197 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:46.195854 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:46.195861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:46.195863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:46.195984 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:34:46.196354 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:34:46.196358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:34:46.196360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:34:46.196707 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:34:46.196970 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 14:34:46.197303 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 14:34:46.197530 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:34:46.198472 31334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:46.261448 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:46.260998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39199 (local address 127.0.0.1:53298)
0810 14:34:46.261196 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:46.261284 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 14:34:46.262498 31370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:46.262115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39199 (local address 127.0.0.1:53312)
0810 14:34:46.262318 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:46.262390 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 14:34:46.284929 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 31115
I20260810 14:34:46.311206 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 30467
I20260810 14:34:46.312294 30590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:34:46.422632 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 30467
I20260810 14:34:46.453749 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36421
--webserver_interface=127.6.46.254
--webserver_port=37119
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:36421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:34:46.775992 31422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:46.776273 31422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:46.776412 31422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:46.783375 31422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:34:46.783483 31422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:46.783558 31422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:46.783638 31422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:34:46.783697 31422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:34:46.793118 31422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:36421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=37119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:46.794391 31422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:46.795660 31422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:46.806715 31435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:46.806674 31436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:46.806775 31438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:46.807494 31422 server_base.cc:1034] running on GCE node
I20260810 14:34:46.808184 31422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:46.809427 31422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:46.810627 31422 hybrid_clock.cc:648] HybridClock initialized: now 1786372486810557 us; error 49 us; skew 500 ppm
I20260810 14:34:46.810935 31422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:46 dist-test-slave-q9h9 krb5kdc[29487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372486, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:46.813916 31422 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:34:46.815292 31422 webserver.cc:460] Webserver started at http://127.6.46.254:37119/ using document root <none> and password file <none>
I20260810 14:34:46.815955 31422 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:46.816079 31422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:46.821148 31422 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 14:34:46.823796 31454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:46.825021 31422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:34:46.825176 31422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "5ba0ce8dcb514d43a6e262bade39824b"
format_stamp: "Formatted at 2026-08-10 14:34:41 on dist-test-slave-q9h9"
server_key: "8592f20b3b8694f5bdace1191b1a95c8"
server_key_iv: "57ce9bc0010938ea1c4efb6c1e45841e"
server_key_version: "encryptionkey@0"
I20260810 14:34:46.826248 31422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:46.842275 31422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:46.845983 31422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:46.846290 31422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:46.863595 31422 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:36421
I20260810 14:34:46.863601 31533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:36421 every 8 connection(s)
I20260810 14:34:46.864980 31422 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:34:46.867199 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 31422
I20260810 14:34:46.868229 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:39199
--local_ip_for_outbound_sockets=127.6.46.193
--tserver_master_addrs=127.6.46.254:36421
--webserver_port=45147
--webserver_interface=127.6.46.193
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:34:46.873411 31536 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:34:46.877807 31536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Bootstrap starting.
I20260810 14:34:46.897994 31536 log.cc:824] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:34:46.914557 31536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:34:46.915110 31536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b: Bootstrap complete.
I20260810 14:34:46.924022 31536 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } }
I20260810 14:34:46.924898 31536 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba0ce8dcb514d43a6e262bade39824b, State: Initialized, Role: FOLLOWER
I20260810 14:34:46.925503 31536 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } }
I20260810 14:34:46.925712 31536 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:34:46.925830 31536 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:34:46.925998 31536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 2 FOLLOWER]: Advancing to term 3
I20260810 14:34:46.929850 31536 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } }
I20260810 14:34:46.930387 31536 leader_election.cc:302] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ba0ce8dcb514d43a6e262bade39824b; no voters:
I20260810 14:34:46.931038 31536 leader_election.cc:288] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 14:34:46.931177 31546 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 3 FOLLOWER]: Leader election won for term 3
I20260810 14:34:46.933135 31536 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:34:46.934305 31546 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [term 3 LEADER]: Becoming Leader. State: Replica: 5ba0ce8dcb514d43a6e262bade39824b, State: Running, Role: LEADER
I20260810 14:34:46.934893 31546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } }
I20260810 14:34:46.937853 31552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ba0ce8dcb514d43a6e262bade39824b. Latest consensus state: current_term: 3 leader_uuid: "5ba0ce8dcb514d43a6e262bade39824b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } } }
I20260810 14:34:46.938717 31552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:46.938042 31549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5ba0ce8dcb514d43a6e262bade39824b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ba0ce8dcb514d43a6e262bade39824b" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36421 } } }
I20260810 14:34:46.939061 31549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ba0ce8dcb514d43a6e262bade39824b [sys.catalog]: This master's current role is: LEADER
I20260810 14:34:46.949770 31560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:34:46.958920 31560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:34:46.960564 31560 catalog_manager.cc:1259] Loaded cluster ID: 1d2671a6a66142a99aa77f7ff4e3e09e
I20260810 14:34:46.960748 31560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 14:34:46.963111 31560 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd8f05fe9ad google::LogMessage::Fail() at ??:0
@ 0x7fd8f060188a google::LogMessage::SendToLog() at ??:0
@ 0x7fd8f05fe4de google::LogMessage::Flush() at ??:0
@ 0x7fd8f05ffa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd906031512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fd90602e976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd9060b6610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fd9060b6439 std::_Function_handler<>::_M_invoke() at ??:0
I20260810 14:34:46.996004 31578 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd8f324a345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd8f324d91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fd8f324d799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fd8f3228eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd8f65026db start_thread at ??:0
@ 0x7fd8ef49861f clone at ??:0
W20260810 14:34:47.228085 31541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:47.228359 31541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:47.228502 31541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:47.235878 31541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:47.236003 31541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:47.236168 31541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:34:47.245770 31541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:39199
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=45147
--tserver_master_addrs=127.6.46.254:36421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:47.246970 31541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:47.248354 31541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:47.260774 31617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:47.260735 31618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:47.260812 31620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:47.261480 31541 server_base.cc:1034] running on GCE node
I20260810 14:34:47.262056 31541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:47.262916 31541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:47.264108 31541 hybrid_clock.cc:648] HybridClock initialized: now 1786372487264055 us; error 43 us; skew 500 ppm
I20260810 14:34:47.264392 31541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:47 dist-test-slave-q9h9 krb5kdc[29487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372487, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:47.267477 31541 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:34:47.268888 31541 webserver.cc:460] Webserver started at http://127.6.46.193:45147/ using document root <none> and password file <none>
I20260810 14:34:47.269761 31541 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:47.269927 31541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:47.275034 31541 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:34:47.277948 31634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:47.279191 31541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:34:47.279337 31541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "0439ed7a9a8d4b638990bc35aab79b5d"
format_stamp: "Formatted at 2026-08-10 14:34:41 on dist-test-slave-q9h9"
server_key: "527d8a2d364685638ec345407cf7ddf2"
server_key_iv: "8fc9d5212a52c812af2dcad56db5ce65"
server_key_version: "encryptionkey@0"
I20260810 14:34:47.280258 31541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:47.303794 31541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:47.307226 31541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:47.307456 31541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:47.308377 31541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:47.309618 31541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:47.309715 31541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:47.309821 31541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:47.309894 31541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:47.340634 31541 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:39199
I20260810 14:34:47.340700 31863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:39199 every 8 connection(s)
I20260810 14:34:47.342000 31541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:34:47.346769 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:47.344691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:35525)
0810 14:34:47.345569 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:47.345876 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":326,"threads_started":1}
W20260810 14:34:47.347386 31864 heartbeater.cc:644] Failed to heartbeat to 127.6.46.254:36421 (0 consecutive failures): Network error: Failed to ping master at 127.6.46.254:36421: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
I20260810 14:34:47.347595 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 31541
I20260810 14:34:47.348179 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:47.347872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:46871)
0810 14:34:47.348005 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:47.348069 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 14:34:47.348358 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:35761
--local_ip_for_outbound_sockets=127.6.46.194
--tserver_master_addrs=127.6.46.254:36421
--webserver_port=43071
--webserver_interface=127.6.46.194
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:34:47.349046 31866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:47.348740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:48821)
0810 14:34:47.348869 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:47.348941 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 14:34:47.349293 31864 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:34:47.672803 31867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:47.673049 31867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:47.673182 31867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:47.680182 31867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:47.680280 31867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:47.680423 31867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:34:47.689627 31867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:35761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=43071
--tserver_master_addrs=127.6.46.254:36421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:47.690805 31867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:47.692029 31867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:47.703940 31876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:47.703986 31879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:47.704080 31875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:47.704913 31867 server_base.cc:1034] running on GCE node
I20260810 14:34:47.705446 31867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:47.706322 31867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:47.707502 31867 hybrid_clock.cc:648] HybridClock initialized: now 1786372487707454 us; error 46 us; skew 500 ppm
I20260810 14:34:47.707756 31867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:47 dist-test-slave-q9h9 krb5kdc[29487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372487, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:47.710651 31867 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:34:47.711959 31867 webserver.cc:460] Webserver started at http://127.6.46.194:43071/ using document root <none> and password file <none>
I20260810 14:34:47.712602 31867 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:47.712716 31867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:47.717680 31867 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:34:47.720372 31894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:47.721607 31867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:34:47.721750 31867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "5f190e4521e24479b6f81542daf7ed8f"
format_stamp: "Formatted at 2026-08-10 14:34:42 on dist-test-slave-q9h9"
server_key: "3a5c921c678055a371552c0f6cb96961"
server_key_iv: "ed4a923b73ff9bfc5db2319572d2560d"
server_key_version: "encryptionkey@0"
I20260810 14:34:47.722702 31867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:47.759433 31867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:47.763176 31867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:47.763428 31867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:47.764349 31867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:47.765586 31867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:47.765692 31867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:47.765801 31867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:47.765879 31867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:47.795732 31867 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:35761
I20260810 14:34:47.795794 32109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:35761 every 8 connection(s)
I20260810 14:34:47.797096 31867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:34:47.801450 32113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:47.799723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:43085)
0810 14:34:47.800426 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:47.800633 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":265,"threads_started":1}
W20260810 14:34:47.802151 32110 heartbeater.cc:644] Failed to heartbeat to 127.6.46.254:36421 (0 consecutive failures): Network error: Failed to ping master at 127.6.46.254:36421: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
I20260810 14:34:47.802851 32113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:47.802566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:55699)
0810 14:34:47.802690 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:47.802754 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 14:34:47.803267 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 31867
I20260810 14:34:47.803653 32113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:47.803346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:38867)
0810 14:34:47.803483 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:47.803555 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 14:34:47.803871 32110 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 14:34:47.803937 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:41189
--local_ip_for_outbound_sockets=127.6.46.195
--tserver_master_addrs=127.6.46.254:36421
--webserver_port=45457
--webserver_interface=127.6.46.195
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:34:48.120007 32118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:34:48.120242 32118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:34:48.120340 32118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:34:48.127225 32118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:34:48.127317 32118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:34:48.127429 32118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:34:48.136471 32118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:41189
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=45457
--tserver_master_addrs=127.6.46.254:36421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:34:48.137542 32118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:34:48.138760 32118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:34:48.150866 32129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:48.150929 32132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:34:48.150902 32128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:34:48.151482 32118 server_base.cc:1034] running on GCE node
I20260810 14:34:48.152010 32118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:34:48.152850 32118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:34:48.154070 32118 hybrid_clock.cc:648] HybridClock initialized: now 1786372488153996 us; error 35 us; skew 500 ppm
I20260810 14:34:48.154361 32118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:34:48 dist-test-slave-q9h9 krb5kdc[29487](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372488, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:34:48.157223 32118 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:34:48.158565 32118 webserver.cc:460] Webserver started at http://127.6.46.195:45457/ using document root <none> and password file <none>
I20260810 14:34:48.159168 32118 fs_manager.cc:359] Metadata directory not provided
I20260810 14:34:48.159286 32118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:34:48.164201 32118 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 14:34:48.166833 32143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:48.168138 32118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:34:48.168306 32118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "5c81f4b07e7d40128063aa51b3bee2fb"
format_stamp: "Formatted at 2026-08-10 14:34:42 on dist-test-slave-q9h9"
server_key: "90b840cdfcd8d008fcada18af56b7409"
server_key_iv: "6c95082d7fe58b0bc4c2e46a8c44b01a"
server_key_version: "encryptionkey@0"
I20260810 14:34:48.169193 32118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:34:48.184039 32118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:34:48.188023 32118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:34:48.188278 32118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:34:48.189174 32118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:34:48.190462 32118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:34:48.190557 32118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:48.190658 32118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:34:48.190730 32118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:34:48.221204 32118 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:41189
I20260810 14:34:48.221266 32332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:41189 every 8 connection(s)
I20260810 14:34:48.222651 32118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:34:48.227497 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:48.225623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:37621)
0810 14:34:48.226399 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:48.226595 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":317,"threads_started":1}
W20260810 14:34:48.228116 32335 heartbeater.cc:644] Failed to heartbeat to 127.6.46.254:36421 (0 consecutive failures): Network error: Failed to ping master at 127.6.46.254:36421: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
I20260810 14:34:48.228571 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 32118
I20260810 14:34:48.228957 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:48.228558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:33609)
0810 14:34:48.228745 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:48.228829 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 14:34:48.229761 32339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:48.229452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:55441)
0810 14:34:48.229587 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:48.229659 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 14:34:48.230046 32335 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 14:34:48.350535 32341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:48.349781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:58505)
0810 14:34:48.350234 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:48.350319 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":195,"threads_started":1}
I20260810 14:34:48.805256 32342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:48.804366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:56405)
0810 14:34:48.804852 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:48.804971 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":226,"threads_started":1}
I20260810 14:34:49.231216 32343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:49.230493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:34039)
0810 14:34:49.230906 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:49.231005 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":204,"threads_started":1}
I20260810 14:34:49.351963 32346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:49.351296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:46751)
0810 14:34:49.351730 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:49.351806 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":218,"threads_started":1}
I20260810 14:34:49.806888 32347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:49.806165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:50449)
0810 14:34:49.806644 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:49.806726 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":235,"threads_started":1}
I20260810 14:34:50.232635 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:50.231976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:51293)
0810 14:34:50.232406 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:50.232484 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":216,"threads_started":1}
I20260810 14:34:50.353452 32349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:50.352775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:56779)
0810 14:34:50.353200 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:50.353277 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":213,"threads_started":1}
I20260810 14:34:50.808466 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:50.807787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:55845)
0810 14:34:50.808228 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:50.808305 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":250,"threads_started":1}
I20260810 14:34:51.234264 32351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:51.233481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:49375)
0810 14:34:51.233942 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:51.234088 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":250,"threads_started":1}
I20260810 14:34:51.354933 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:51.354240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:60043)
0810 14:34:51.354664 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:51.354751 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":208,"threads_started":1}
I20260810 14:34:51.809937 32354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:51.809273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:60639)
0810 14:34:51.809707 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:51.809783 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":226,"threads_started":1}
I20260810 14:34:52.235764 32355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:52.235099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:60427)
0810 14:34:52.235502 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:52.235576 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":200,"threads_started":1}
I20260810 14:34:52.356490 32356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:52.355773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:58709)
0810 14:34:52.356243 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:52.356322 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":232,"threads_started":1}
I20260810 14:34:52.811419 32357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:52.810748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:46395)
0810 14:34:52.811175 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:52.811257 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":223,"threads_started":1}
I20260810 14:34:53.237344 32359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:53.236623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:46401)
0810 14:34:53.237101 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:53.237180 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":258,"threads_started":1}
I20260810 14:34:53.357970 32361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:53.357312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:42889)
0810 14:34:53.357725 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:53.357813 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":201,"threads_started":1}
I20260810 14:34:53.812873 32364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:53.812212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:41973)
0810 14:34:53.812637 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:53.812721 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":209,"threads_started":1}
I20260810 14:34:54.238863 32365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:54.238173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:53125)
0810 14:34:54.238629 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:54.238713 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":231,"threads_started":1}
I20260810 14:34:54.359691 32366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:54.358836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:51761)
0810 14:34:54.359368 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:54.359462 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":227,"threads_started":1}
I20260810 14:34:54.814414 32369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:54.813646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:34275)
0810 14:34:54.814138 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:54.814213 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":213,"threads_started":1}
I20260810 14:34:55.240561 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:55.239752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:50619)
0810 14:34:55.240243 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:55.240326 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":217,"threads_started":1}
I20260810 14:34:55.361055 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:55.360443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:53083)
0810 14:34:55.360827 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:55.360900 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":198,"threads_started":1}
I20260810 14:34:55.815922 32372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:55.815218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:46115)
0810 14:34:55.815670 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:55.815750 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":238,"threads_started":1}
I20260810 14:34:56.242136 32373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:56.241410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:49205)
0810 14:34:56.241824 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:56.241900 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":217,"threads_started":1}
I20260810 14:34:56.362519 32374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:56.361822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:35995)
0810 14:34:56.362280 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:56.362358 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":239,"threads_started":1}
I20260810 14:34:56.817701 32375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:56.816910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:57985)
0810 14:34:56.817406 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:56.817508 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":237,"threads_started":1}
I20260810 14:34:57.243799 32376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:57.243045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:56651)
0810 14:34:57.243534 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:57.243631 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":235,"threads_started":1}
I20260810 14:34:57.364003 32380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:57.363324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:36053)
0810 14:34:57.363769 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:57.363849 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":227,"threads_started":1}
I20260810 14:34:57.819157 32381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:57.818508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:44973)
0810 14:34:57.818915 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:57.818990 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":219,"threads_started":1}
I20260810 14:34:58.245262 32383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:58.244591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:33805)
0810 14:34:58.245028 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:58.245105 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":235,"threads_started":1}
I20260810 14:34:58.365417 32384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:58.364764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:43949)
0810 14:34:58.365164 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:58.365252 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":197,"threads_started":1}
I20260810 14:34:58.820653 32385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:58.819956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:46843)
0810 14:34:58.820391 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:58.820478 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":223,"threads_started":1}
I20260810 14:34:59.246793 32386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:59.246063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:46631)
0810 14:34:59.246518 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:59.246612 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":220,"threads_started":1}
I20260810 14:34:59.366858 32387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:59.366215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:56849)
0810 14:34:59.366636 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:59.366712 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":212,"threads_started":1}
I20260810 14:34:59.822154 32388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:34:59.821487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:49447)
0810 14:34:59.821892 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0810 14:34:59.821967 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":207,"threads_started":1}
I20260810 14:35:00.248310 32389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:00.247641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:48299)
0810 14:35:00.248082 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:00.248158 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":220,"threads_started":1}
I20260810 14:35:00.368541 32390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:00.367659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:55479)
0810 14:35:00.368248 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:00.368353 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":269,"threads_started":1}
I20260810 14:35:00.823617 32391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:00.822923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:46309)
0810 14:35:00.823346 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:00.823437 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":210,"threads_started":1}
I20260810 14:35:01.249960 32394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:01.249154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:39413)
0810 14:35:01.249677 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:01.249772 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":263,"threads_started":1}
I20260810 14:35:01.370160 32395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:01.369402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:47887)
0810 14:35:01.369873 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:01.369957 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":266,"threads_started":1}
I20260810 14:35:01.825071 32396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:01.824397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:45435)
0810 14:35:01.824812 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:01.824890 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":213,"threads_started":1}
I20260810 14:35:02.251489 32397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:02.250802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:35555)
0810 14:35:02.251233 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:02.251312 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":212,"threads_started":1}
I20260810 14:35:02.371889 32398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:02.371148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.193:47853)
0810 14:35:02.371581 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:02.371693 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":218,"threads_started":1}
I20260810 14:35:02.826973 32399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:02.826269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.194:55165)
0810 14:35:02.826696 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:02.826789 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":214,"threads_started":1}
I20260810 14:35:03.239482 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 31541
I20260810 14:35:03.253232 32402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:03.252469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36421 (local address 127.6.46.195:53953)
0810 14:35:03.252948 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:03.253040 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:36421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":272,"threads_started":1}
I20260810 14:35:03.261418 32400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:03.240514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.6.46.193:39199)
0810 14:35:03.241180 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:03.241198 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:03.241246 (+ 48us) server_negotiation.cc:407] Connection header received
0810 14:35:03.242854 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:03.242864 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:03.243291 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:03.243570 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:03.245006 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:03.246296 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:03.247528 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:03.248162 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:03.248292 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:03.255489 (+ 7197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:03.255496 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:03.255510 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:03.255544 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:03.258315 (+ 2771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:03.258968 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:03.258974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:03.258980 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:03.259148 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:03.259518 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:03.259521 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:03.259523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:03.260017 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:03.260289 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:03.260744 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 14:35:03.261145 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":505,"threads_started":1}
I20260810 14:35:03.262773 31852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:35:03.347688 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 31541
I20260810 14:35:03.368201 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 31867
I20260810 14:35:03.385928 32408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:03.369113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39540 (local address 127.6.46.194:35761)
0810 14:35:03.369646 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:03.369660 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:03.369696 (+ 36us) server_negotiation.cc:407] Connection header received
0810 14:35:03.371378 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:03.371387 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:03.371814 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:03.372116 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:03.374210 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:03.375427 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:03.376827 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:03.377472 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:03.377601 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:03.380075 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:03.380082 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:03.380096 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:03.380133 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:03.383294 (+ 3161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:03.383886 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:03.383891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:03.383897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:03.384056 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:03.384331 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:03.384334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:03.384336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:03.384811 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:03.385030 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:03.385340 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 14:35:03.385639 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":236,"threads_started":1}
I20260810 14:35:03.387199 32103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:35:03.468199 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 31867
I20260810 14:35:03.490988 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 32118
I20260810 14:35:03.507954 32410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:03.492253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.6.46.195:41189)
0810 14:35:03.492782 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:03.492796 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:03.492842 (+ 46us) server_negotiation.cc:407] Connection header received
0810 14:35:03.494562 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:03.494572 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:03.495019 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:03.495305 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:03.496486 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:03.497688 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:03.498986 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:03.499618 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:03.499766 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:03.502196 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:03.502204 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:03.502218 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:03.502258 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:03.505017 (+ 2759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:03.505624 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:03.505630 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:03.505635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:03.505804 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:03.506207 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:03.506210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:03.506212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:03.506705 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:03.506936 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:03.507291 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 14:35:03.507659 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":225,"threads_started":1}
I20260810 14:35:03.509377 32323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:35:03.590131 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 32118
2026-08-10T14:35:03Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24196 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:35:03 dist-test-slave-q9h9 krb5kdc[32420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:35:03 dist-test-slave-q9h9 krb5kdc[32420](info): set up 2 sockets
Aug 10 14:35:03 dist-test-slave-q9h9 krb5kdc[32420](info): commencing operation
krb5kdc: starting...
W20260810 14:35:04.819025 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.184s user 0.001s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:35:04 dist-test-slave-q9h9 krb5kdc[32420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:35:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:35:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:35:05.060436 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:41101
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:41101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:05.374341 32438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:05.374619 32438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:05.374768 32438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:05.381584 32438 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 14:35:05.381681 32438 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 14:35:05.381745 32438 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 14:35:05.381816 32438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:05.381888 32438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:05.381978 32438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:05.382062 32438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:05.382139 32438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:05.391405 32438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:41101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:41101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:35:05.392513 32438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:05.393690 32438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:05.404784 32446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:05.404810 32447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:05.404924 32450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:05.405499 32438 server_base.cc:1034] running on GCE node
I20260810 14:35:05.406275 32438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:05.407509 32438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:05.408730 32438 hybrid_clock.cc:648] HybridClock initialized: now 1786372505408667 us; error 38 us; skew 500 ppm
I20260810 14:35:05.409030 32438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:35:05 dist-test-slave-q9h9 krb5kdc[32420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372505, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:05.411949 32438 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:35:05.413230 32438 webserver.cc:460] Webserver started at http://127.6.46.254:35593/ using document root <none> and password file <none>
I20260810 14:35:05.413848 32438 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:05.413949 32438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:05.414273 32438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:05.417027 32438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "4aab9c7489ad4f0fb41f5c240aca6b04"
format_stamp: "Formatted at 2026-08-10 14:35:05 on dist-test-slave-q9h9"
server_key: "d429999aa21d50299bd329d770bed149"
server_key_iv: "27b2ade9c96ef4de34a3ad4af1ee3bee"
server_key_version: "encryptionkey@0"
I20260810 14:35:05.417765 32438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "4aab9c7489ad4f0fb41f5c240aca6b04"
format_stamp: "Formatted at 2026-08-10 14:35:05 on dist-test-slave-q9h9"
server_key: "d429999aa21d50299bd329d770bed149"
server_key_iv: "27b2ade9c96ef4de34a3ad4af1ee3bee"
server_key_version: "encryptionkey@0"
I20260810 14:35:05.422534 32438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 14:35:05.425689 32463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:05.426891 32438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:35:05.427047 32438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "4aab9c7489ad4f0fb41f5c240aca6b04"
format_stamp: "Formatted at 2026-08-10 14:35:05 on dist-test-slave-q9h9"
server_key: "d429999aa21d50299bd329d770bed149"
server_key_iv: "27b2ade9c96ef4de34a3ad4af1ee3bee"
server_key_version: "encryptionkey@0"
I20260810 14:35:05.427243 32438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:05.448436 32438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:05.452402 32438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:05.452631 32438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:05.469698 32438 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:41101
I20260810 14:35:05.469720 32558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:41101 every 8 connection(s)
I20260810 14:35:05.470995 32438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:05.474429 32559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:05.479557 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 32438
I20260810 14:35:05.479770 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:35:05.480611 6331 external_mini_cluster.cc:1428] Setting key fe03b3b088377a03b1f903fd5a94fb63
I20260810 14:35:05.485229 32559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Bootstrap starting.
I20260810 14:35:05.488639 32559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:05.489814 32559 log.cc:824] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:05.492815 32559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: No bootstrap required, opened a new log
Aug 10 14:35:05 dist-test-slave-q9h9 krb5kdc[32420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:35:05.501806 32559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } }
I20260810 14:35:05.502189 32559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:05.502300 32559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aab9c7489ad4f0fb41f5c240aca6b04, State: Initialized, Role: FOLLOWER
I20260810 14:35:05.502959 32559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } }
I20260810 14:35:05.503247 32559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:05.503383 32559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:05.503572 32559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:05.504530 32564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:05.482665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32996 (local address 127.6.46.254:41101)
0810 14:35:05.483463 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:05.483474 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:05.483514 (+ 40us) server_negotiation.cc:407] Connection header received
0810 14:35:05.485117 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:05.485145 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:05.485555 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:05.485865 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:05.487614 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:05.488936 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:05.490757 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:05.491522 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:05.491643 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:05.496826 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:05.496836 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:05.496855 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:05.496907 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:05.500180 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:05.500853 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:05.500860 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:05.500865 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:05.501094 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:05.501464 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:05.501470 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:05.501473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:05.501974 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:05.502301 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:05.502770 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 14:35:05.503198 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":375,"threads_started":1}
I20260810 14:35:05.506186 32559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } }
I20260810 14:35:05.506637 32559 leader_election.cc:302] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aab9c7489ad4f0fb41f5c240aca6b04; no voters:
I20260810 14:35:05.507231 32559 leader_election.cc:288] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:35:05.507328 32566 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:05.507997 32566 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 1 LEADER]: Becoming Leader. State: Replica: 4aab9c7489ad4f0fb41f5c240aca6b04, State: Running, Role: LEADER
I20260810 14:35:05.508512 32566 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } }
I20260810 14:35:05.509156 32559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:05.510885 32569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aab9c7489ad4f0fb41f5c240aca6b04. Latest consensus state: current_term: 1 leader_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } } }
I20260810 14:35:05.510977 32567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } } }
I20260810 14:35:05.511381 32569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:05.511500 32567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:05.512953 32575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:05.517143 32575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:05.525504 32575 catalog_manager.cc:1347] Generated new cluster ID: 11c56c5a0c894737b87adfe1736be1ba
I20260810 14:35:05.525599 32575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:05.552820 32575 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:35:05.553964 32575 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:05.564015 32575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Generated new TSK 0
I20260810 14:35:05.564689 32575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:35:05.628650 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:41101
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:05.963263 32607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:05.963521 32607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:05.963651 32607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:05.970453 32607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:05.970549 32607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:05.970703 32607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:35:05.979750 32607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:35:05.980863 32607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:05.982095 32607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:05.994294 32616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:05.994428 32613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:05.994318 32614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:05.995107 32607 server_base.cc:1034] running on GCE node
I20260810 14:35:05.995672 32607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:05.996512 32607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:05.997704 32607 hybrid_clock.cc:648] HybridClock initialized: now 1786372505997652 us; error 38 us; skew 500 ppm
I20260810 14:35:05.997980 32607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:35:05 dist-test-slave-q9h9 krb5kdc[32420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372505, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:06.000916 32607 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:35:06.002250 32607 webserver.cc:460] Webserver started at http://127.6.46.193:41173/ using document root <none> and password file <none>
I20260810 14:35:06.002833 32607 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:06.002929 32607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:06.003170 32607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:06.005955 32607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "abb955bb82774b849f4e8f922af12e51"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "6b814150aa85b6ddc6ecde301692321a"
server_key_iv: "64fe1906d0e1a87825e10293bc24c67e"
server_key_version: "encryptionkey@0"
I20260810 14:35:06.006703 32607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "abb955bb82774b849f4e8f922af12e51"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "6b814150aa85b6ddc6ecde301692321a"
server_key_iv: "64fe1906d0e1a87825e10293bc24c67e"
server_key_version: "encryptionkey@0"
I20260810 14:35:06.011472 32607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:06.014649 32628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:06.015847 32607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 14:35:06.015980 32607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "abb955bb82774b849f4e8f922af12e51"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "6b814150aa85b6ddc6ecde301692321a"
server_key_iv: "64fe1906d0e1a87825e10293bc24c67e"
server_key_version: "encryptionkey@0"
I20260810 14:35:06.016144 32607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:06.034813 32607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:06.038522 32607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:06.038739 32607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:06.039556 32607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:06.040680 32607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:06.040748 32607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:06.040834 32607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:06.040876 32607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:06.068996 32607 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:38933
I20260810 14:35:06.069051 358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:38933 every 8 connection(s)
I20260810 14:35:06.070142 32607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:06.074949 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 32607
I20260810 14:35:06.075173 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:35:06.076037 6331 external_mini_cluster.cc:1428] Setting key 41ab6b7a80af9cf7ecc6f41a3cb81830
Aug 10 14:35:06 dist-test-slave-q9h9 krb5kdc[32420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372505, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:35:06.088831 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:06.072342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:42077 (local address 127.6.46.254:41101)
0810 14:35:06.072749 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:06.072754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:06.073531 (+ 777us) server_negotiation.cc:407] Connection header received
0810 14:35:06.075647 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:06.075651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:06.075718 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:06.075810 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:06.077534 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:06.078355 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:06.079821 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:06.080208 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:06.080319 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:06.083742 (+ 3423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:06.083747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:06.083750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:06.083780 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:06.086092 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:06.086851 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:06.086855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:06.086857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:06.086977 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:06.087377 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:06.087380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:06.087382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:06.087607 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:06.087721 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:06.088402 (+ 681us) server_negotiation.cc:299] Negotiation successful
0810 14:35:06.088558 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":210,"threads_started":1}
I20260810 14:35:06.089885 363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:06.072704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:42077)
0810 14:35:06.073361 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:06.073395 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:06.075347 (+ 1952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:06.076069 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:06.076080 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:06.076744 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:06.077370 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:06.077388 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:06.078506 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:06.078510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:06.079687 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:06.079699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:06.080424 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:06.080431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:06.080635 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:06.081219 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:06.081244 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:06.083532 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:06.086229 (+ 2697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:06.086237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:06.086249 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:06.086754 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:06.087071 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:06.087075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:06.087078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:06.087292 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:06.087723 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:06.087730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:06.088068 (+ 338us) client_negotiation.cc:769] Sending connection context
0810 14:35:06.088335 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 14:35:06.088769 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":253,"threads_started":1}
I20260810 14:35:06.091230 359 heartbeater.cc:347] Connected to a master server at 127.6.46.254:41101
I20260810 14:35:06.091507 359 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:06.092259 359 heartbeater.cc:511] Master 127.6.46.254:41101 requested a full tablet report, sending...
I20260810 14:35:06.094290 32497 ts_manager.cc:194] Registered new tserver with Master: abb955bb82774b849f4e8f922af12e51 (127.6.46.193:38933)
I20260810 14:35:06.095669 32497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:42077
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:35:06.134649 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:41101
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:06.445538 369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:06.445781 369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:06.445899 369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:06.452893 369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:06.452993 369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:06.453153 369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:35:06.462507 369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:35:06.463660 369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:06.464915 369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:06.478768 385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:06.478760 382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:06.479058 381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:06.480265 369 server_base.cc:1034] running on GCE node
I20260810 14:35:06.480845 369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:06.481736 369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:06.482949 369 hybrid_clock.cc:648] HybridClock initialized: now 1786372506482900 us; error 44 us; skew 500 ppm
I20260810 14:35:06.483230 369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:35:06 dist-test-slave-q9h9 krb5kdc[32420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372506, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:06.486212 369 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:35:06.487599 369 webserver.cc:460] Webserver started at http://127.6.46.194:35991/ using document root <none> and password file <none>
I20260810 14:35:06.488222 369 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:06.488332 369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:06.488627 369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:06.491417 369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "f531c50d23964a129cb7c5ce591b1335"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "831ab8b045f379a9de6dde5d0c721f96"
server_key_iv: "619efde213a57ffd10a2fe6898d502e0"
server_key_version: "encryptionkey@0"
I20260810 14:35:06.492126 369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "f531c50d23964a129cb7c5ce591b1335"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "831ab8b045f379a9de6dde5d0c721f96"
server_key_iv: "619efde213a57ffd10a2fe6898d502e0"
server_key_version: "encryptionkey@0"
I20260810 14:35:06.497020 369 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 14:35:06.501003 398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:06.502369 369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 14:35:06.502498 369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "f531c50d23964a129cb7c5ce591b1335"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "831ab8b045f379a9de6dde5d0c721f96"
server_key_iv: "619efde213a57ffd10a2fe6898d502e0"
server_key_version: "encryptionkey@0"
I20260810 14:35:06.502666 369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:06.522706 369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:06.526566 369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:06.526824 369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:06.527770 369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:06.529088 369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:06.529175 369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:06.529294 369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:06.529345 369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:06.558686 369 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:45835
I20260810 14:35:06.558753 593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:45835 every 8 connection(s)
I20260810 14:35:06.559937 369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:06.565711 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 369
I20260810 14:35:06.565922 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:35:06.566520 6331 external_mini_cluster.cc:1428] Setting key a930929a6fd95383f447f477265835bc
Aug 10 14:35:06 dist-test-slave-q9h9 krb5kdc[32420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372506, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:35:06.580109 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:06.562370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:41525 (local address 127.6.46.254:41101)
0810 14:35:06.562590 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:06.562596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:06.563704 (+ 1108us) server_negotiation.cc:407] Connection header received
0810 14:35:06.565504 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:06.565508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:06.565578 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:06.565665 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:06.567559 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:06.568285 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:06.570410 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:06.570837 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:06.570933 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:06.574741 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:06.574746 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:06.574749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:06.574782 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:06.577147 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:06.577944 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:06.577948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:06.577950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:06.578135 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:06.578554 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:06.578557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:06.578559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:06.578806 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:06.578912 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:06.579684 (+ 772us) server_negotiation.cc:299] Negotiation successful
0810 14:35:06.579859 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:35:06.581197 600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:06.562845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:41525)
0810 14:35:06.563532 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:06.563580 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:06.565209 (+ 1629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:06.565945 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:06.565957 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:06.566697 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:06.567385 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:06.567405 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:06.568437 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:06.568442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:06.570268 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:06.570284 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:06.571061 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:06.571068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:06.571344 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:06.572002 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:06.572034 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:06.574518 (+ 2484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:06.577276 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:06.577285 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:06.577298 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:06.577820 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:06.578233 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:06.578237 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:06.578241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:06.578460 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:06.578917 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:06.578924 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:06.579294 (+ 370us) client_negotiation.cc:769] Sending connection context
0810 14:35:06.579568 (+ 274us) client_negotiation.cc:240] Negotiation successful
0810 14:35:06.580041 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":261,"threads_started":1}
I20260810 14:35:06.582501 596 heartbeater.cc:347] Connected to a master server at 127.6.46.254:41101
I20260810 14:35:06.582769 596 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:06.583436 596 heartbeater.cc:511] Master 127.6.46.254:41101 requested a full tablet report, sending...
I20260810 14:35:06.584793 32497 ts_manager.cc:194] Registered new tserver with Master: f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835)
I20260810 14:35:06.585559 32497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:41525
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:35:06.623121 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:41101
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:06.934914 603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:06.935173 603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:06.935314 603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:06.942160 603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:06.942266 603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:06.942421 603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:35:06.951512 603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:35:06.952704 603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:06.953946 603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:06.966087 615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:06.966140 613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:06.966341 608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:06.967056 603 server_base.cc:1034] running on GCE node
I20260810 14:35:06.967585 603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:06.968438 603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:06.969623 603 hybrid_clock.cc:648] HybridClock initialized: now 1786372506969558 us; error 59 us; skew 500 ppm
I20260810 14:35:06.969879 603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:35:06 dist-test-slave-q9h9 krb5kdc[32420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372506, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:06.972749 603 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:35:06.974069 603 webserver.cc:460] Webserver started at http://127.6.46.195:34151/ using document root <none> and password file <none>
I20260810 14:35:06.974709 603 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:06.974843 603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:06.975121 603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:35:06.977818 603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "07cc4f5ef6fa4877a026847e535b87eb"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "6804b4ef75e56e88264d941889208a2e"
server_key_iv: "7dae886e82cdc3d0350892af32034d39"
server_key_version: "encryptionkey@0"
I20260810 14:35:06.978574 603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "07cc4f5ef6fa4877a026847e535b87eb"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "6804b4ef75e56e88264d941889208a2e"
server_key_iv: "7dae886e82cdc3d0350892af32034d39"
server_key_version: "encryptionkey@0"
I20260810 14:35:06.983266 603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:35:06.986488 626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:06.987648 603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:35:06.987794 603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "07cc4f5ef6fa4877a026847e535b87eb"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "6804b4ef75e56e88264d941889208a2e"
server_key_iv: "7dae886e82cdc3d0350892af32034d39"
server_key_version: "encryptionkey@0"
I20260810 14:35:06.988027 603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:07.003158 603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:07.006915 603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:07.007133 603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:07.007956 603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:07.009203 603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:07.009275 603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:07.009363 603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:07.009407 603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:07.039489 603 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:42925
I20260810 14:35:07.039531 806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:42925 every 8 connection(s)
I20260810 14:35:07.040654 603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:07.044225 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 603
I20260810 14:35:07.044418 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:35:07.045063 6331 external_mini_cluster.cc:1428] Setting key 422e9ec55fcf44a20c67be32a30aa004
Aug 10 14:35:07 dist-test-slave-q9h9 krb5kdc[32420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372506, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:35:07.059973 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.043043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:50363 (local address 127.6.46.254:41101)
0810 14:35:07.043223 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:07.043228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:07.044377 (+ 1149us) server_negotiation.cc:407] Connection header received
0810 14:35:07.046639 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:07.046643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:07.046708 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:07.046783 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:07.048542 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.049284 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.050761 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.051161 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.051252 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.054325 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:07.054330 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:07.054334 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:07.054369 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:07.057122 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:07.057729 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:07.057733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:07.057735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:07.057886 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:07.058278 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:07.058282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:07.058284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:07.058575 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:07.058726 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:07.059387 (+ 661us) server_negotiation.cc:299] Negotiation successful
0810 14:35:07.059606 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:35:07.060786 809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.043405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:50363)
0810 14:35:07.044198 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:07.044259 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:07.046380 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:07.047027 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:07.047035 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:07.047683 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:07.048386 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:07.048411 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.049408 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.049411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:07.050627 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:07.050640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.051320 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.051325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:07.051524 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.052067 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:07.052088 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:07.054110 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:07.057226 (+ 3116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:07.057231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:07.057241 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:07.057605 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:07.057971 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:07.057974 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:07.057976 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:07.058183 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:07.058704 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:07.058710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:07.058988 (+ 278us) client_negotiation.cc:769] Sending connection context
0810 14:35:07.059267 (+ 279us) client_negotiation.cc:240] Negotiation successful
0810 14:35:07.059733 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":296,"threads_started":1}
I20260810 14:35:07.062105 807 heartbeater.cc:347] Connected to a master server at 127.6.46.254:41101
I20260810 14:35:07.062389 807 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:07.063108 807 heartbeater.cc:511] Master 127.6.46.254:41101 requested a full tablet report, sending...
I20260810 14:35:07.064515 32497 ts_manager.cc:194] Registered new tserver with Master: 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925)
I20260810 14:35:07.065311 32497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:50363
I20260810 14:35:07.072481 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:35:07.089347 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.076882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.6.46.254:41101)
0810 14:35:07.077038 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:07.077044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:07.077438 (+ 394us) server_negotiation.cc:407] Connection header received
0810 14:35:07.077622 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:07.077628 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:07.077697 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:07.077778 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:07.079363 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.080097 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.081516 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.081949 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.082112 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.084517 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:07.084523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:07.084526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:07.084557 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:07.086903 (+ 2346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:07.087600 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:07.087606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:07.087610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:07.087745 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:07.088174 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:07.088179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:07.088183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:07.088484 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:07.088677 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:07.088968 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 14:35:07.089119 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:35:07.098637 359 heartbeater.cc:503] Master 127.6.46.254:41101 was elected leader, sending a full tablet report...
I20260810 14:35:07.093247 32497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:35:07.100625 32497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:07.130937 833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.113464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.6.46.193:38933)
0810 14:35:07.122497 (+ 9033us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:07.122507 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:07.124004 (+ 1497us) server_negotiation.cc:407] Connection header received
0810 14:35:07.124095 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:07.124102 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:07.124323 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:07.124515 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:07.125422 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.126709 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.128852 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.129836 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.129955 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.130203 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:07.130455 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 14:35:07.130611 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8864,"thread_start_us":8941,"threads_started":1}
I20260810 14:35:07.131052 828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.112499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:38933 (local address 127.0.0.1:59694)
0810 14:35:07.123766 (+ 11267us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:07.123780 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:07.123893 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:07.124774 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:07.124779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:07.124803 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:07.125229 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:07.125242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.126836 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.126842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:07.128664 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:07.128678 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.130044 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.130052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:07.130245 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.130288 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 14:35:07.130391 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 14:35:07.130616 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11168,"thread_start_us":8315,"threads_started":1}
I20260810 14:35:07.131670 830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.111923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:42925 (local address 127.0.0.1:53974)
0810 14:35:07.121236 (+ 9313us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:07.121282 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:07.121462 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:07.123898 (+ 2436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:07.123905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:07.123931 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:07.124324 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:07.124338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.126299 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.126303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:07.128850 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:07.128865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.130897 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.130903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:07.131061 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.131089 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 14:35:07.131176 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 14:35:07.131290 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9031,"mutex_wait_us":128,"thread_start_us":11264,"threads_started":1}
I20260810 14:35:07.131829 832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.113038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.6.46.194:45835)
0810 14:35:07.122032 (+ 8994us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:07.122042 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:07.123758 (+ 1716us) server_negotiation.cc:407] Connection header received
0810 14:35:07.123870 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:07.123876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:07.124083 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:07.124260 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:07.124823 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.126104 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.129033 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.130427 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.131064 (+ 637us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.131251 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:07.131323 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 14:35:07.131472 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8804,"thread_start_us":8700,"threads_started":1}
I20260810 14:35:07.132066 825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.112200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:45835 (local address 127.0.0.1:33648)
0810 14:35:07.123462 (+ 11262us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:07.123478 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:07.123618 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:07.124269 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:07.124273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:07.124297 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:07.124637 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:07.124650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.126239 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.126244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:07.128852 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:07.128865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.130685 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.130691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:07.130879 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.130911 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 14:35:07.130983 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 14:35:07.131676 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11123,"mutex_wait_us":628,"thread_start_us":11327,"threads_started":1}
I20260810 14:35:07.132668 829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.112241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.6.46.195:42925)
0810 14:35:07.123217 (+ 10976us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:07.123225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:07.123258 (+ 33us) server_negotiation.cc:407] Connection header received
0810 14:35:07.123352 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:07.123360 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:07.123596 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:07.123876 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:07.124553 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.126182 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.129033 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.130423 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.130546 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.130763 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:07.132208 (+ 1445us) server_negotiation.cc:299] Negotiation successful
0810 14:35:07.132360 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10805,"thread_start_us":10654,"threads_started":1}
I20260810 14:35:07.135197 32720 tablet_service.cc:1467] Processing CreateTablet for tablet 69765dedefe3414388f06261c19bb73a (DEFAULT_TABLE table=test-table [id=2487d627d52a4b4abd87d5f347ebbfd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:07.135944 705 tablet_service.cc:1467] Processing CreateTablet for tablet 69765dedefe3414388f06261c19bb73a (DEFAULT_TABLE table=test-table [id=2487d627d52a4b4abd87d5f347ebbfd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:07.135949 483 tablet_service.cc:1467] Processing CreateTablet for tablet 69765dedefe3414388f06261c19bb73a (DEFAULT_TABLE table=test-table [id=2487d627d52a4b4abd87d5f347ebbfd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:07.136847 32720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69765dedefe3414388f06261c19bb73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:07.137143 705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69765dedefe3414388f06261c19bb73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:07.137466 483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69765dedefe3414388f06261c19bb73a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:07.147771 835 tablet_bootstrap.cc:492] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb: Bootstrap starting.
I20260810 14:35:07.150758 836 tablet_bootstrap.cc:492] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51: Bootstrap starting.
I20260810 14:35:07.151957 835 tablet_bootstrap.cc:654] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:07.152223 838 tablet_bootstrap.cc:492] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335: Bootstrap starting.
I20260810 14:35:07.153297 835 log.cc:824] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:07.154202 836 tablet_bootstrap.cc:654] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:07.155299 838 tablet_bootstrap.cc:654] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:07.155303 836 log.cc:824] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:07.156342 838 log.cc:824] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:07.157115 835 tablet_bootstrap.cc:492] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb: No bootstrap required, opened a new log
I20260810 14:35:07.157439 835 ts_tablet_manager.cc:1397] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260810 14:35:07.159085 838 tablet_bootstrap.cc:492] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335: No bootstrap required, opened a new log
I20260810 14:35:07.159339 838 ts_tablet_manager.cc:1397] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 14:35:07.160408 836 tablet_bootstrap.cc:492] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51: No bootstrap required, opened a new log
I20260810 14:35:07.160714 836 ts_tablet_manager.cc:1397] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260810 14:35:07.166450 838 raft_consensus.cc:348] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } }
I20260810 14:35:07.166792 838 raft_consensus.cc:374] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:07.166908 838 raft_consensus.cc:729] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f531c50d23964a129cb7c5ce591b1335, State: Initialized, Role: FOLLOWER
I20260810 14:35:07.167438 838 consensus_queue.cc:260] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } }
I20260810 14:35:07.167862 835 raft_consensus.cc:348] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } }
I20260810 14:35:07.168426 596 heartbeater.cc:503] Master 127.6.46.254:41101 was elected leader, sending a full tablet report...
I20260810 14:35:07.168879 838 ts_tablet_manager.cc:1428] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20260810 14:35:07.169064 835 raft_consensus.cc:374] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:07.169185 835 raft_consensus.cc:729] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cc4f5ef6fa4877a026847e535b87eb, State: Initialized, Role: FOLLOWER
I20260810 14:35:07.169729 835 consensus_queue.cc:260] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } }
I20260810 14:35:07.170516 836 raft_consensus.cc:348] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } }
I20260810 14:35:07.170765 807 heartbeater.cc:503] Master 127.6.46.254:41101 was elected leader, sending a full tablet report...
I20260810 14:35:07.170979 836 raft_consensus.cc:374] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:07.171136 836 raft_consensus.cc:729] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb955bb82774b849f4e8f922af12e51, State: Initialized, Role: FOLLOWER
I20260810 14:35:07.171409 835 ts_tablet_manager.cc:1428] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb: Time spent starting tablet: real 0.014s user 0.004s sys 0.008s
I20260810 14:35:07.171852 836 consensus_queue.cc:260] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } }
I20260810 14:35:07.174499 836 ts_tablet_manager.cc:1428] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51: Time spent starting tablet: real 0.014s user 0.011s sys 0.001s
W20260810 14:35:07.292169 808 tablet.cc:2369] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:35:07.311758 598 tablet.cc:2369] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:35:07.322209 360 tablet.cc:2369] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:35:07.420416 849 raft_consensus.cc:482] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:07.420696 849 raft_consensus.cc:504] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } }
I20260810 14:35:07.422406 849 leader_election.cc:288] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925), f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835)
I20260810 14:35:07.430838 829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.422720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:56939 (local address 127.6.46.195:42925)
0810 14:35:07.422932 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:07.422938 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:07.423478 (+ 540us) server_negotiation.cc:407] Connection header received
0810 14:35:07.423774 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:07.423780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:07.423865 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:07.423964 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:07.424670 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.425793 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.428695 (+ 2902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.429895 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.430079 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.430172 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:07.430437 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 14:35:07.430568 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:35:07.430882 850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.422666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:42925 (local address 127.6.46.193:56939)
0810 14:35:07.423359 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:07.423379 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:07.423615 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:07.423985 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:07.423988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:07.424007 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:07.424504 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:07.424516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.425924 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.425930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:07.428522 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:07.428538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.430121 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.430126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:07.430286 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.430314 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 14:35:07.430388 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 14:35:07.430477 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":164,"spinlock_wait_cycles":23424,"thread_start_us":333,"threads_started":1}
I20260810 14:35:07.431214 851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.422678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:45835 (local address 127.6.46.193:57779)
0810 14:35:07.423713 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:07.423729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:07.423822 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:07.424129 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:07.424136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:07.424168 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:07.424605 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:07.424615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.425924 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.425930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:07.428522 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:07.428538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.430324 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.430331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:07.430493 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.430529 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 14:35:07.430637 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 14:35:07.430762 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":152,"thread_start_us":682,"threads_started":1}
I20260810 14:35:07.431237 832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.423270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:57779 (local address 127.6.46.194:45835)
0810 14:35:07.423463 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:07.423468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:07.423804 (+ 336us) server_negotiation.cc:407] Connection header received
0810 14:35:07.423927 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:07.423933 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:07.424006 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:07.424093 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:07.424764 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.425794 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.428695 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.429895 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.430079 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.430172 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:07.430686 (+ 514us) server_negotiation.cc:299] Negotiation successful
0810 14:35:07.430817 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 14:35:07.432051 742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69765dedefe3414388f06261c19bb73a" candidate_uuid: "abb955bb82774b849f4e8f922af12e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cc4f5ef6fa4877a026847e535b87eb" is_pre_election: true
I20260810 14:35:07.432066 517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69765dedefe3414388f06261c19bb73a" candidate_uuid: "abb955bb82774b849f4e8f922af12e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f531c50d23964a129cb7c5ce591b1335" is_pre_election: true
I20260810 14:35:07.432533 517 raft_consensus.cc:2454] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb955bb82774b849f4e8f922af12e51 in term 0.
I20260810 14:35:07.432531 742 raft_consensus.cc:2454] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abb955bb82774b849f4e8f922af12e51 in term 0.
I20260810 14:35:07.433171 32632 leader_election.cc:302] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 07cc4f5ef6fa4877a026847e535b87eb, abb955bb82774b849f4e8f922af12e51, f531c50d23964a129cb7c5ce591b1335; no voters:
I20260810 14:35:07.433547 849 raft_consensus.cc:2781] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:07.433692 849 raft_consensus.cc:482] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:07.433786 849 raft_consensus.cc:3037] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:07.435937 849 raft_consensus.cc:504] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } }
I20260810 14:35:07.436523 849 leader_election.cc:288] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 1 election: Requested vote from peers 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925), f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835)
I20260810 14:35:07.436847 742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69765dedefe3414388f06261c19bb73a" candidate_uuid: "abb955bb82774b849f4e8f922af12e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cc4f5ef6fa4877a026847e535b87eb"
I20260810 14:35:07.436933 517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69765dedefe3414388f06261c19bb73a" candidate_uuid: "abb955bb82774b849f4e8f922af12e51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f531c50d23964a129cb7c5ce591b1335"
I20260810 14:35:07.437007 742 raft_consensus.cc:3037] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:07.437131 517 raft_consensus.cc:3037] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:07.439172 742 raft_consensus.cc:2454] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb955bb82774b849f4e8f922af12e51 in term 1.
I20260810 14:35:07.439304 517 raft_consensus.cc:2454] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abb955bb82774b849f4e8f922af12e51 in term 1.
I20260810 14:35:07.439505 32632 leader_election.cc:302] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07cc4f5ef6fa4877a026847e535b87eb, abb955bb82774b849f4e8f922af12e51; no voters:
I20260810 14:35:07.439785 849 raft_consensus.cc:2781] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:07.440205 849 raft_consensus.cc:686] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [term 1 LEADER]: Becoming Leader. State: Replica: abb955bb82774b849f4e8f922af12e51, State: Running, Role: LEADER
I20260810 14:35:07.440673 849 consensus_queue.cc:237] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } }
I20260810 14:35:07.444847 32493 catalog_manager.cc:5495] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 reported cstate change: term changed from 0 to 1, leader changed from <none> to abb955bb82774b849f4e8f922af12e51 (127.6.46.193). New cstate: current_term: 1 leader_uuid: "abb955bb82774b849f4e8f922af12e51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } health_report { overall_health: HEALTHY } } }
I20260810 14:35:07.497682 833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.492997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.6.46.193:38933)
0810 14:35:07.493171 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:07.493176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:07.493193 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:35:07.493336 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:07.493339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:07.493414 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:07.493530 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:07.493821 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.494543 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.495933 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.496327 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.496416 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.496792 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:07.497291 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:07.497401 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:07.497438 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 14:35:07.497499 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:35:07.505851 742 raft_consensus.cc:1261] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb [term 1 FOLLOWER]: Refusing update from remote peer abb955bb82774b849f4e8f922af12e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:07.505954 517 raft_consensus.cc:1261] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335 [term 1 FOLLOWER]: Refusing update from remote peer abb955bb82774b849f4e8f922af12e51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:07.506809 849 consensus_queue.cc:1035] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:07.507083 852 consensus_queue.cc:1035] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:07.519269 859 mvcc.cc:205] Tried to move back new op lower bound from 7316981790734065664 to 7316981790480482304. Current Snapshot: MvccSnapshot[applied={T|T < 7316981790734065664}]
I20260810 14:35:07.529500 32493 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:07.529800 32493 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:07.533367 32493 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Sending DeleteTablet for 3 replicas of tablet 69765dedefe3414388f06261c19bb73a
I20260810 14:35:07.534459 705 tablet_service.cc:1514] Processing DeleteTablet for tablet 69765dedefe3414388f06261c19bb73a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:07 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:53974
I20260810 14:35:07.534493 483 tablet_service.cc:1514] Processing DeleteTablet for tablet 69765dedefe3414388f06261c19bb73a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:07 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:33648
I20260810 14:35:07.534797 32720 tablet_service.cc:1514] Processing DeleteTablet for tablet 69765dedefe3414388f06261c19bb73a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:07 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:59694
I20260810 14:35:07.535353 877 tablet_replica.cc:330] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb: stopping tablet replica
I20260810 14:35:07.535567 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 32607
I20260810 14:35:07.535684 877 raft_consensus.cc:2229] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:07.536113 877 raft_consensus.cc:2258] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:07.536832 873 tablet_replica.cc:330] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335: stopping tablet replica
I20260810 14:35:07.537182 878 tablet_replica.cc:330] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51: stopping tablet replica
I20260810 14:35:07.537206 873 raft_consensus.cc:2229] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:07.537602 878 raft_consensus.cc:2229] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:35:07.537729 873 raft_consensus.cc:2258] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:07.538533 878 raft_consensus.cc:2258] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:07.539460 877 ts_tablet_manager.cc:1905] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:07.540241 873 ts_tablet_manager.cc:1905] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:07.543943 878 ts_tablet_manager.cc:1905] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:07.546828 877 ts_tablet_manager.cc:1918] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:07.547034 877 log.cc:1196] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/69765dedefe3414388f06261c19bb73a
I20260810 14:35:07.547608 877 ts_tablet_manager.cc:1939] T 69765dedefe3414388f06261c19bb73a P 07cc4f5ef6fa4877a026847e535b87eb: Deleting consensus metadata
I20260810 14:35:07.549386 32467 catalog_manager.cc:4841] TS 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925): tablet 69765dedefe3414388f06261c19bb73a (table test-table [id=2487d627d52a4b4abd87d5f347ebbfd9]) successfully deleted
I20260810 14:35:07.551097 878 ts_tablet_manager.cc:1918] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:07.551357 878 log.cc:1196] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/69765dedefe3414388f06261c19bb73a
I20260810 14:35:07.551944 878 ts_tablet_manager.cc:1939] T 69765dedefe3414388f06261c19bb73a P abb955bb82774b849f4e8f922af12e51: Deleting consensus metadata
I20260810 14:35:07.553597 32465 catalog_manager.cc:4841] TS abb955bb82774b849f4e8f922af12e51 (127.6.46.193:38933): tablet 69765dedefe3414388f06261c19bb73a (table test-table [id=2487d627d52a4b4abd87d5f347ebbfd9]) successfully deleted
I20260810 14:35:07.554688 873 ts_tablet_manager.cc:1918] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:07.554877 873 log.cc:1196] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/69765dedefe3414388f06261c19bb73a
I20260810 14:35:07.555387 873 ts_tablet_manager.cc:1939] T 69765dedefe3414388f06261c19bb73a P f531c50d23964a129cb7c5ce591b1335: Deleting consensus metadata
I20260810 14:35:07.556993 32464 catalog_manager.cc:4841] TS f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835): tablet 69765dedefe3414388f06261c19bb73a (table test-table [id=2487d627d52a4b4abd87d5f347ebbfd9]) successfully deleted
Aug 10 14:35:07 dist-test-slave-q9h9 krb5kdc[32420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:35:07.565496 833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.536669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.6.46.193:38933)
0810 14:35:07.536864 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:07.536870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:07.546603 (+ 9733us) server_negotiation.cc:407] Connection header received
0810 14:35:07.546802 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:07.546809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:07.546894 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:07.546999 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:07.549109 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.550386 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.552277 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.552953 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.553051 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.558471 (+ 5420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:07.558484 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:07.558501 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:07.558559 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:07.562414 (+ 3855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:07.563316 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:07.563324 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:07.563328 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:07.563475 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:07.563855 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:07.563860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:07.563864 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:07.564298 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:07.564590 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:07.564900 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 14:35:07.565139 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:35:07.566439 353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:35:07.663833 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 32607
I20260810 14:35:07.686924 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 369
Aug 10 14:35:07 dist-test-slave-q9h9 krb5kdc[32420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:35:07.701342 832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.687631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33662 (local address 127.6.46.194:45835)
0810 14:35:07.687814 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:07.687821 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:07.687841 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:35:07.688015 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:07.688020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:07.688087 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:07.688174 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:07.689408 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.690143 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.691489 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.691892 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.691996 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.696405 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:07.696418 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:07.696433 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:07.696464 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:07.698947 (+ 2483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:07.699702 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:07.699707 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:07.699710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:07.699829 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:07.700111 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:07.700114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:07.700115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:07.700450 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:07.700679 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:07.700933 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 14:35:07.701081 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:35:07.701947 587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:35:07.787423 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 369
I20260810 14:35:07.808977 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 603
Aug 10 14:35:07 dist-test-slave-q9h9 krb5kdc[32420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372504, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:35:07.823158 829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:07.809726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.6.46.195:42925)
0810 14:35:07.809881 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:07.809885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:07.809904 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:35:07.810075 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:07.810080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:07.810163 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:07.810259 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:07.811643 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.812353 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.813621 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:07.813982 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:07.814124 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:07.818374 (+ 4250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:07.818389 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:07.818406 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:07.818438 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:07.820797 (+ 2359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:07.821410 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:07.821417 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:07.821419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:07.821539 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:07.821813 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:07.821816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:07.821818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:07.822300 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:07.822508 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:07.822749 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 14:35:07.822897 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:35:07.823791 799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:35:07.910048 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 603
I20260810 14:35:07.931881 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 32438
I20260810 14:35:07.932969 32554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:35:08.039201 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 32438
I20260810 14:35:28.066308 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:41101
--webserver_interface=127.6.46.254
--webserver_port=35593
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:41101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:28.403592 888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:28.403853 888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:28.403990 888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:28.411083 888 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 14:35:28.411167 888 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 14:35:28.411212 888 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 14:35:28.411263 888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:28.411315 888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:28.411381 888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:28.411425 888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:28.411476 888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:28.420915 888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:41101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:41101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=35593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:35:28.422092 888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:28.423327 888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:28.447060 897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:28.447127 895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:28.447106 903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:28.447386 888 server_base.cc:1034] running on GCE node
I20260810 14:35:28.448431 888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:28.449931 888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:28.451161 888 hybrid_clock.cc:648] HybridClock initialized: now 1786372528451096 us; error 34 us; skew 500 ppm
I20260810 14:35:28.451527 888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:35:28 dist-test-slave-q9h9 krb5kdc[32420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372528, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:28.455062 888 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:35:28.456599 888 webserver.cc:460] Webserver started at http://127.6.46.254:35593/ using document root <none> and password file <none>
I20260810 14:35:28.457319 888 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:28.457440 888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:28.463191 888 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:35:28.466048 915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:28.467362 888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:35:28.467531 888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "4aab9c7489ad4f0fb41f5c240aca6b04"
format_stamp: "Formatted at 2026-08-10 14:35:05 on dist-test-slave-q9h9"
server_key: "d429999aa21d50299bd329d770bed149"
server_key_iv: "27b2ade9c96ef4de34a3ad4af1ee3bee"
server_key_version: "encryptionkey@0"
I20260810 14:35:28.468511 888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:28.506294 888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:28.510152 888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:28.510413 888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:28.528324 888 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:41101
I20260810 14:35:28.528368 990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:41101 every 8 connection(s)
I20260810 14:35:28.529798 888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:28.536211 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 888
I20260810 14:35:28.536738 992 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:35:28.537030 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:38933
--local_ip_for_outbound_sockets=127.6.46.193
--tserver_master_addrs=127.6.46.254:41101
--webserver_port=41173
--webserver_interface=127.6.46.193
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:35:28.540017 992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Bootstrap starting.
I20260810 14:35:28.559983 992 log.cc:824] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:28.572952 992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:35:28.573470 992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Bootstrap complete.
I20260810 14:35:28.582469 992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } }
I20260810 14:35:28.583336 992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aab9c7489ad4f0fb41f5c240aca6b04, State: Initialized, Role: FOLLOWER
I20260810 14:35:28.583945 992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } }
I20260810 14:35:28.584173 992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:28.584280 992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:28.584434 992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 1 FOLLOWER]: Advancing to term 2
I20260810 14:35:28.588117 992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } }
I20260810 14:35:28.588613 992 leader_election.cc:302] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aab9c7489ad4f0fb41f5c240aca6b04; no voters:
I20260810 14:35:28.589203 992 leader_election.cc:288] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 14:35:28.589334 998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 14:35:28.590512 998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 2 LEADER]: Becoming Leader. State: Replica: 4aab9c7489ad4f0fb41f5c240aca6b04, State: Running, Role: LEADER
I20260810 14:35:28.591135 998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } }
I20260810 14:35:28.591419 992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:28.593591 1000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } } }
I20260810 14:35:28.593959 1000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:28.597690 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aab9c7489ad4f0fb41f5c240aca6b04. Latest consensus state: current_term: 2 leader_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } } }
I20260810 14:35:28.598107 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:28.599649 1006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:28.607576 1006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:28.609011 1006 catalog_manager.cc:1259] Loaded cluster ID: 11c56c5a0c894737b87adfe1736be1ba
I20260810 14:35:28.609153 1006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:28.613004 1006 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:28.614269 1026 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 14:35:28.614589 1006 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Loaded TSK: 0
I20260810 14:35:28.645313 1006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Generated new TSK 1
I20260810 14:35:28.645722 1006 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 14:35:28.873450 996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:28.873754 996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:28.873891 996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:28.881058 996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:28.881170 996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:28.881318 996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:35:28.890619 996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:38933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=41173
--tserver_master_addrs=127.6.46.254:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:35:28.891837 996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:28.893141 996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:28.907418 1033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:28.907792 1034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:28.908053 1037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:28.908627 996 server_base.cc:1034] running on GCE node
I20260810 14:35:28.909165 996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:28.910010 996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:28.911232 996 hybrid_clock.cc:648] HybridClock initialized: now 1786372528911175 us; error 54 us; skew 500 ppm
I20260810 14:35:28.911509 996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:35:28 dist-test-slave-q9h9 krb5kdc[32420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372528, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:28.914508 996 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:35:28.915886 996 webserver.cc:460] Webserver started at http://127.6.46.193:41173/ using document root <none> and password file <none>
I20260810 14:35:28.916502 996 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:28.916632 996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:28.921778 996 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:28.924530 1050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:28.925896 996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:28.926096 996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "abb955bb82774b849f4e8f922af12e51"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "6b814150aa85b6ddc6ecde301692321a"
server_key_iv: "64fe1906d0e1a87825e10293bc24c67e"
server_key_version: "encryptionkey@0"
I20260810 14:35:28.927186 996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:28.943676 996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:28.947690 996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:28.947934 996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:28.948948 996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:28.950295 996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:28.950397 996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:28.950496 996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:28.950568 996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:28.980468 996 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:38933
I20260810 14:35:28.980515 1223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:38933 every 8 connection(s)
I20260810 14:35:28.981806 996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:28.983731 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 996
I20260810 14:35:28.984458 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:45835
--local_ip_for_outbound_sockets=127.6.46.194
--tserver_master_addrs=127.6.46.254:41101
--webserver_port=35991
--webserver_interface=127.6.46.194
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:35:28 dist-test-slave-q9h9 krb5kdc[32420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372528, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:35:29.005280 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:28.984682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:45743 (local address 127.6.46.254:41101)
0810 14:35:28.985293 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:28.985308 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:28.985875 (+ 567us) server_negotiation.cc:407] Connection header received
0810 14:35:28.988034 (+ 2159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:28.988050 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:28.988464 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:28.988856 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:28.990175 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.991988 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.993595 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.994535 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.994740 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:28.998189 (+ 3449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:28.998197 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:28.998213 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:28.998249 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:29.001151 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.002053 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.002060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.002065 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.002231 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.002664 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.002667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.002669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.003112 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:29.003250 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.004068 (+ 818us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.004375 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":239,"threads_started":1}
I20260810 14:35:29.005546 1234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:28.984794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:45743)
0810 14:35:28.985694 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:28.985738 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:28.987472 (+ 1734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:28.988970 (+ 1498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:28.988977 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:28.989450 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:28.989971 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:28.989988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.992146 (+ 2158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.992150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:28.993455 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:28.993468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:28.994749 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:28.994754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:28.994951 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:28.995561 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:28.995584 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:28.997905 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:29.001298 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.001307 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.001320 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.001910 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.002335 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.002340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.002344 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.002571 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.003255 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:29.003263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:29.003632 (+ 369us) client_negotiation.cc:769] Sending connection context
0810 14:35:29.003902 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 14:35:29.004408 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":401,"threads_started":1}
I20260810 14:35:29.008601 1226 heartbeater.cc:347] Connected to a master server at 127.6.46.254:41101
I20260810 14:35:29.008877 1226 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:29.009590 1226 heartbeater.cc:511] Master 127.6.46.254:41101 requested a full tablet report, sending...
I20260810 14:35:29.012003 938 ts_manager.cc:194] Registered new tserver with Master: abb955bb82774b849f4e8f922af12e51 (127.6.46.193:38933)
I20260810 14:35:29.014185 938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:45743
W20260810 14:35:29.315214 1235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:29.315469 1235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:29.315562 1235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:29.322544 1235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:29.322644 1235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:29.322767 1235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:35:29.332257 1235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:45835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=35991
--tserver_master_addrs=127.6.46.254:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:35:29.333489 1235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:29.334780 1235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:29.347282 1242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:29.347260 1244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:29.347220 1246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:29.348384 1235 server_base.cc:1034] running on GCE node
I20260810 14:35:29.348941 1235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:29.349992 1235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:29.351222 1235 hybrid_clock.cc:648] HybridClock initialized: now 1786372529351169 us; error 44 us; skew 500 ppm
I20260810 14:35:29.351518 1235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:35:29 dist-test-slave-q9h9 krb5kdc[32420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372529, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:29.354465 1235 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:35:29.355815 1235 webserver.cc:460] Webserver started at http://127.6.46.194:35991/ using document root <none> and password file <none>
I20260810 14:35:29.356427 1235 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:29.356544 1235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:29.361570 1235 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:29.364245 1259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:29.365694 1235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 14:35:29.365824 1235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "f531c50d23964a129cb7c5ce591b1335"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "831ab8b045f379a9de6dde5d0c721f96"
server_key_iv: "619efde213a57ffd10a2fe6898d502e0"
server_key_version: "encryptionkey@0"
I20260810 14:35:29.366789 1235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:29.388115 1235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:29.392616 1235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:29.392843 1235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:29.393694 1235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:29.395138 1235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:29.395247 1235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:29.395327 1235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:29.395388 1235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:29.423935 1235 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:45835
I20260810 14:35:29.423957 1441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:45835 every 8 connection(s)
I20260810 14:35:29.425370 1235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:29.435591 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 1235
I20260810 14:35:29.436208 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:42925
--local_ip_for_outbound_sockets=127.6.46.195
--tserver_master_addrs=127.6.46.254:41101
--webserver_port=34151
--webserver_interface=127.6.46.195
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 14:35:29 dist-test-slave-q9h9 krb5kdc[32420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372529, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:35:29.444581 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.427814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:47437 (local address 127.6.46.254:41101)
0810 14:35:29.428024 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.428029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.429154 (+ 1125us) server_negotiation.cc:407] Connection header received
0810 14:35:29.431131 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.431137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.431218 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.431306 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:29.432808 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.433550 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.435176 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.435600 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.435712 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.439191 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:29.439196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:29.439200 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:29.439234 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:29.441624 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.442367 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.442371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.442373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.442502 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.442919 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.442923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.442925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.443215 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:29.443352 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.444010 (+ 658us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.444255 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:35:29.445489 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.428258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:47437)
0810 14:35:29.428984 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:29.429033 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:29.430852 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:29.431561 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:29.431569 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:29.432075 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:29.432632 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.432651 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.433711 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.433715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.435031 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.435045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.435833 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.435842 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.436067 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.436678 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:29.436705 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:29.438966 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:29.441759 (+ 2793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.441768 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.441783 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.442242 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.442615 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.442620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.442623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.442803 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.443343 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:29.443350 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:29.443660 (+ 310us) client_negotiation.cc:769] Sending connection context
0810 14:35:29.443889 (+ 229us) client_negotiation.cc:240] Negotiation successful
0810 14:35:29.444370 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":286,"threads_started":1}
I20260810 14:35:29.446894 1447 heartbeater.cc:347] Connected to a master server at 127.6.46.254:41101
I20260810 14:35:29.447175 1447 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:29.447855 1447 heartbeater.cc:511] Master 127.6.46.254:41101 requested a full tablet report, sending...
I20260810 14:35:29.449295 938 ts_manager.cc:194] Registered new tserver with Master: f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835)
I20260810 14:35:29.450322 938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:47437
W20260810 14:35:29.768105 1450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:29.768364 1450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:29.768456 1450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:29.775712 1450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:29.775810 1450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:29.775924 1450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:35:29.785102 1450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:42925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=34151
--tserver_master_addrs=127.6.46.254:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:35:29.786269 1450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:29.787465 1450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:29.800127 1463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:29.800084 1464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:29.800155 1466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:29.800783 1450 server_base.cc:1034] running on GCE node
I20260810 14:35:29.801363 1450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:29.802286 1450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:29.803505 1450 hybrid_clock.cc:648] HybridClock initialized: now 1786372529803445 us; error 53 us; skew 500 ppm
I20260810 14:35:29.803810 1450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:35:29 dist-test-slave-q9h9 krb5kdc[32420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372529, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:29.806840 1450 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:35:29.808301 1450 webserver.cc:460] Webserver started at http://127.6.46.195:34151/ using document root <none> and password file <none>
I20260810 14:35:29.808924 1450 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:29.809048 1450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:29.814178 1450 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:35:29.816843 1478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:29.818147 1450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:29.818302 1450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "07cc4f5ef6fa4877a026847e535b87eb"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "6804b4ef75e56e88264d941889208a2e"
server_key_iv: "7dae886e82cdc3d0350892af32034d39"
server_key_version: "encryptionkey@0"
I20260810 14:35:29.819231 1450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:29.856323 1450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:29.860349 1450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:29.860590 1450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:29.861582 1450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:29.862895 1450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:29.862999 1450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:29.863098 1450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:29.863166 1450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:29.892704 1450 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:42925
I20260810 14:35:29.892786 1651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:42925 every 8 connection(s)
I20260810 14:35:29.894091 1450 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:29.894965 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 1450
Aug 10 14:35:29 dist-test-slave-q9h9 krb5kdc[32420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372529, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:35:29.909936 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.895982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57522 (local address 127.6.46.254:41101)
0810 14:35:29.896155 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.896161 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.896717 (+ 556us) server_negotiation.cc:407] Connection header received
0810 14:35:29.896909 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.896914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.897000 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.897116 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:29.898755 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.899481 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.901085 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.901574 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.901688 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.904409 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:29.904414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:29.904420 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:29.904451 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:29.907268 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.907872 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.907877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.907881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.908010 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.908370 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.908379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.908382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.908750 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:29.909073 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.909342 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.909570 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:35:29.913856 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.896748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:52731 (local address 127.6.46.254:41101)
0810 14:35:29.898253 (+ 1505us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.898259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.898278 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:35:29.900084 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.900090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.900165 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.900269 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:29.901767 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.902646 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.904119 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.904548 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.904657 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.907952 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:29.907957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:29.907960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:29.907987 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:29.910835 (+ 2848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.911622 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.911626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.911628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.911728 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.912201 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.912206 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.912209 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.912458 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:29.912581 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.913273 (+ 692us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.913481 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1384,"spinlock_wait_cycles":896,"thread_start_us":236,"threads_started":1}
I20260810 14:35:29.914745 1661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.897300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:52731)
0810 14:35:29.898114 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:29.898156 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:29.899832 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:29.900473 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:29.900480 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:29.901011 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:29.901589 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.901606 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.902788 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.902792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.903985 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.903997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.904766 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.904774 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.904986 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.905479 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:29.905499 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:29.907748 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:29.910953 (+ 3205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.910961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.910975 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.911525 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.911826 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:29.911830 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:29.911834 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:29.912096 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:29.912582 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:29.912589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:29.912910 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 14:35:29.913148 (+ 238us) client_negotiation.cc:240] Negotiation successful
0810 14:35:29.913612 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":301,"threads_started":1}
I20260810 14:35:29.915938 1653 heartbeater.cc:347] Connected to a master server at 127.6.46.254:41101
I20260810 14:35:29.916246 1653 heartbeater.cc:464] Registering TS with master...
I20260810 14:35:29.916921 1653 heartbeater.cc:511] Master 127.6.46.254:41101 requested a full tablet report, sending...
I20260810 14:35:29.918285 938 ts_manager.cc:194] Registered new tserver with Master: 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925)
I20260810 14:35:29.919067 938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:52731
I20260810 14:35:29.924479 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:35:29.930858 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.925336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57538 (local address 127.6.46.254:41101)
0810 14:35:29.925477 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.925481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.925907 (+ 426us) server_negotiation.cc:407] Connection header received
0810 14:35:29.926095 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.926099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.926230 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.926346 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:29.926746 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.927529 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.928972 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.929381 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.929476 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.929971 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:29.930374 (+ 403us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 14:35:29.930672 (+ 298us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57538: token expired
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 14:35:29.930959 1657 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57538: token expired
W20260810 14:35:29.931524 810 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.46.254:41101)
I20260810 14:35:29.943986 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.932028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.6.46.254:41101)
0810 14:35:29.932195 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.932200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.932216 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:35:29.932343 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.932347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.932399 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.932474 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:29.933859 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.934655 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.935940 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.936341 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.936442 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.938978 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:29.938983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:29.938986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:29.939014 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:29.941408 (+ 2394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.942071 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.942076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.942080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.942212 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:29.942563 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:29.942568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:29.942572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:29.942959 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:29.943220 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.943491 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.943689 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:35:29.947363 938 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:35:29.954519 938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:35:29.989148 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.965722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:42925 (local address 127.0.0.1:45432)
0810 14:35:29.980193 (+ 14471us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:29.980231 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:29.980411 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:29.981510 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:29.981517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:29.981547 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:29.981929 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.981942 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.983973 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.983978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.986417 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.986433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.988213 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.988219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.988417 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.988470 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 14:35:29.988589 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 14:35:29.988786 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14123,"thread_start_us":15547,"threads_started":1}
I20260810 14:35:29.989224 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.965918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.6.46.195:42925)
0810 14:35:29.976907 (+ 10989us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.976917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.980411 (+ 3494us) server_negotiation.cc:407] Connection header received
0810 14:35:29.980681 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.980690 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.980954 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.981138 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:29.982182 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.983794 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.986600 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.987997 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.988129 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.988372 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.988688 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.988865 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10803,"thread_start_us":10636,"threads_started":1}
I20260810 14:35:29.989892 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.965766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:38933 (local address 127.0.0.1:40226)
0810 14:35:29.981007 (+ 15241us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:29.981024 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:29.981147 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:29.982194 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:29.982198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:29.982223 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:29.982616 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.982629 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.984757 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.984763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.986995 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.987010 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.989067 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.989073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.989249 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.989284 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 14:35:29.989362 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:35:29.989496 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15108,"mutex_wait_us":2,"thread_start_us":21165,"threads_started":1}
I20260810 14:35:29.990074 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.966113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40226 (local address 127.6.46.193:38933)
0810 14:35:29.978826 (+ 12713us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.978835 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.981287 (+ 2452us) server_negotiation.cc:407] Connection header received
0810 14:35:29.981472 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.981496 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.981727 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.981911 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:29.982852 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.984569 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.987198 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.988571 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.988703 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.988905 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.989575 (+ 670us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.989732 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12542,"thread_start_us":12709,"threads_started":1}
I20260810 14:35:29.991207 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.967621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:45835 (local address 127.0.0.1:40624)
0810 14:35:29.981128 (+ 13507us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:29.981146 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:29.981519 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:29.982444 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:29.982452 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:29.982477 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:29.982908 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.982920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.984826 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.984832 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.987524 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:29.987536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.990403 (+ 2867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.990414 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:29.990582 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.990612 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 14:35:29.990706 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 14:35:29.990828 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13381,"mutex_wait_us":481,"thread_start_us":12084,"threads_started":1}
I20260810 14:35:29.991212 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:29.968233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40624 (local address 127.6.46.194:45835)
0810 14:35:29.979523 (+ 11290us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:29.979532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:29.981266 (+ 1734us) server_negotiation.cc:407] Connection header received
0810 14:35:29.981737 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:29.981745 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:29.981933 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:29.982127 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:29.983097 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.984645 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.987701 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:29.989410 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:29.989528 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:29.989996 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:29.990753 (+ 757us) server_negotiation.cc:299] Negotiation successful
0810 14:35:29.990905 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11143,"thread_start_us":11009,"threads_started":1}
I20260810 14:35:29.993794 1546 tablet_service.cc:1467] Processing CreateTablet for tablet d5c411f7cd3a491dbeceb409a9c7908a (DEFAULT_TABLE table=test-table [id=fa4fbb8c625e4e19b5cf4dfbfe8425f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:29.994426 1349 tablet_service.cc:1467] Processing CreateTablet for tablet d5c411f7cd3a491dbeceb409a9c7908a (DEFAULT_TABLE table=test-table [id=fa4fbb8c625e4e19b5cf4dfbfe8425f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:29.994428 1120 tablet_service.cc:1467] Processing CreateTablet for tablet d5c411f7cd3a491dbeceb409a9c7908a (DEFAULT_TABLE table=test-table [id=fa4fbb8c625e4e19b5cf4dfbfe8425f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:35:29.995529 1546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c411f7cd3a491dbeceb409a9c7908a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:29.995718 1349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c411f7cd3a491dbeceb409a9c7908a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:29.995777 1120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5c411f7cd3a491dbeceb409a9c7908a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:35:30.007579 1676 tablet_bootstrap.cc:492] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335: Bootstrap starting.
I20260810 14:35:30.010443 1677 tablet_bootstrap.cc:492] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51: Bootstrap starting.
I20260810 14:35:30.011248 1680 tablet_bootstrap.cc:492] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb: Bootstrap starting.
I20260810 14:35:30.012197 1676 tablet_bootstrap.cc:654] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:30.013492 1676 log.cc:824] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:30.013953 1677 tablet_bootstrap.cc:654] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:30.014568 1680 tablet_bootstrap.cc:654] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb: Neither blocks nor log segments found. Creating new log.
I20260810 14:35:30.015290 1677 log.cc:824] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:30.015570 1680 log.cc:824] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:30.016752 1676 tablet_bootstrap.cc:492] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335: No bootstrap required, opened a new log
I20260810 14:35:30.017030 1676 ts_tablet_manager.cc:1397] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260810 14:35:30.017252 1226 heartbeater.cc:503] Master 127.6.46.254:41101 was elected leader, sending a full tablet report...
I20260810 14:35:30.018285 1677 tablet_bootstrap.cc:492] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51: No bootstrap required, opened a new log
I20260810 14:35:30.018528 1680 tablet_bootstrap.cc:492] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb: No bootstrap required, opened a new log
I20260810 14:35:30.018530 1677 ts_tablet_manager.cc:1397] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 14:35:30.018759 1680 ts_tablet_manager.cc:1397] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 14:35:30.025928 1676 raft_consensus.cc:348] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:30.026312 1676 raft_consensus.cc:374] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:30.026415 1676 raft_consensus.cc:729] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f531c50d23964a129cb7c5ce591b1335, State: Initialized, Role: FOLLOWER
I20260810 14:35:30.026997 1676 consensus_queue.cc:260] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:30.028784 1677 raft_consensus.cc:348] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:30.028784 1680 raft_consensus.cc:348] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:30.029188 1680 raft_consensus.cc:374] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:30.029179 1677 raft_consensus.cc:374] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:35:30.029281 1677 raft_consensus.cc:729] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb955bb82774b849f4e8f922af12e51, State: Initialized, Role: FOLLOWER
I20260810 14:35:30.029282 1680 raft_consensus.cc:729] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07cc4f5ef6fa4877a026847e535b87eb, State: Initialized, Role: FOLLOWER
I20260810 14:35:30.029846 1677 consensus_queue.cc:260] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:30.029846 1680 consensus_queue.cc:260] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:30.031605 1653 heartbeater.cc:503] Master 127.6.46.254:41101 was elected leader, sending a full tablet report...
I20260810 14:35:30.031970 1677 ts_tablet_manager.cc:1428] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260810 14:35:30.032982 1680 ts_tablet_manager.cc:1428] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 14:35:30.034291 1447 heartbeater.cc:503] Master 127.6.46.254:41101 was elected leader, sending a full tablet report...
I20260810 14:35:30.035275 1676 ts_tablet_manager.cc:1428] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
I20260810 14:35:30.039167 1688 raft_consensus.cc:482] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:30.039358 1688 raft_consensus.cc:504] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:30.040772 1688 leader_election.cc:288] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925), abb955bb82774b849f4e8f922af12e51 (127.6.46.193:38933)
I20260810 14:35:30.047057 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.041473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:56351 (local address 127.6.46.193:38933)
0810 14:35:30.041655 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.041662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.041793 (+ 131us) server_negotiation.cc:407] Connection header received
0810 14:35:30.042085 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.042089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.042148 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.042237 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:30.042855 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.043784 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.045447 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.046239 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.046377 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.046450 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.046710 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.046833 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:35:30.047165 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.041052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:38933 (local address 127.6.46.194:56351)
0810 14:35:30.041673 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.041702 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:30.041905 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:30.042284 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:30.042289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:30.042320 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:30.042702 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.042716 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.043908 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.043913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.045305 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.045317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.046436 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.046442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.046567 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.046591 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:35:30.046672 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 14:35:30.046804 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":512,"threads_started":1}
I20260810 14:35:30.047556 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.041517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:56485 (local address 127.6.46.195:42925)
0810 14:35:30.041804 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.041808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.042580 (+ 772us) server_negotiation.cc:407] Connection header received
0810 14:35:30.042706 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.042710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.042780 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.042856 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:35:30.043423 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.044271 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.045911 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.046694 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.046839 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.046907 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.047239 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.047368 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260810 14:35:30.048048 1695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.041069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:42925 (local address 127.6.46.194:56485)
0810 14:35:30.042485 (+ 1416us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.042498 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:30.042597 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:30.042894 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:30.042898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:30.042923 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:35:30.043282 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.043293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.044390 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.044395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.045765 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:30.045777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.046900 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.046907 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:30.047059 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.047081 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 14:35:30.047719 (+ 638us) client_negotiation.cc:240] Negotiation successful
0810 14:35:30.047832 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"thread_start_us":1107,"threads_started":1}
I20260810 14:35:30.048072 1148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c411f7cd3a491dbeceb409a9c7908a" candidate_uuid: "f531c50d23964a129cb7c5ce591b1335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb955bb82774b849f4e8f922af12e51" is_pre_election: true
I20260810 14:35:30.048590 1148 raft_consensus.cc:2454] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f531c50d23964a129cb7c5ce591b1335 in term 0.
I20260810 14:35:30.049012 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c411f7cd3a491dbeceb409a9c7908a" candidate_uuid: "f531c50d23964a129cb7c5ce591b1335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cc4f5ef6fa4877a026847e535b87eb" is_pre_election: true
I20260810 14:35:30.049269 1264 leader_election.cc:302] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abb955bb82774b849f4e8f922af12e51, f531c50d23964a129cb7c5ce591b1335; no voters:
I20260810 14:35:30.049435 1579 raft_consensus.cc:2454] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f531c50d23964a129cb7c5ce591b1335 in term 0.
I20260810 14:35:30.049659 1688 raft_consensus.cc:2781] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:35:30.049810 1688 raft_consensus.cc:482] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:35:30.049914 1688 raft_consensus.cc:3037] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:30.052079 1688 raft_consensus.cc:504] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:30.052634 1688 leader_election.cc:288] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [CANDIDATE]: Term 1 election: Requested vote from peers 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925), abb955bb82774b849f4e8f922af12e51 (127.6.46.193:38933)
I20260810 14:35:30.052965 1579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c411f7cd3a491dbeceb409a9c7908a" candidate_uuid: "f531c50d23964a129cb7c5ce591b1335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07cc4f5ef6fa4877a026847e535b87eb"
I20260810 14:35:30.053128 1148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5c411f7cd3a491dbeceb409a9c7908a" candidate_uuid: "f531c50d23964a129cb7c5ce591b1335" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abb955bb82774b849f4e8f922af12e51"
I20260810 14:35:30.053162 1579 raft_consensus.cc:3037] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:30.053318 1148 raft_consensus.cc:3037] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:35:30.056344 1579 raft_consensus.cc:2454] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f531c50d23964a129cb7c5ce591b1335 in term 1.
I20260810 14:35:30.056344 1148 raft_consensus.cc:2454] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f531c50d23964a129cb7c5ce591b1335 in term 1.
I20260810 14:35:30.056720 1264 leader_election.cc:302] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abb955bb82774b849f4e8f922af12e51, f531c50d23964a129cb7c5ce591b1335; no voters:
I20260810 14:35:30.057001 1688 raft_consensus.cc:2781] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:35:30.057422 1688 raft_consensus.cc:686] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [term 1 LEADER]: Becoming Leader. State: Replica: f531c50d23964a129cb7c5ce591b1335, State: Running, Role: LEADER
I20260810 14:35:30.057852 1688 consensus_queue.cc:237] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:30.062357 937 catalog_manager.cc:5495] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 reported cstate change: term changed from 0 to 1, leader changed from <none> to f531c50d23964a129cb7c5ce591b1335 (127.6.46.194). New cstate: current_term: 1 leader_uuid: "f531c50d23964a129cb7c5ce591b1335" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } health_report { overall_health: HEALTHY } } }
I20260810 14:35:30.077953 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.073152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.6.46.194:45835)
0810 14:35:30.073311 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.073315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.073331 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:35:30.073441 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.073444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.073513 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.073627 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:35:30.073904 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.074681 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.076152 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.076539 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.076626 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.077030 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:35:30.077479 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:35:30.077586 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.077708 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.077769 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:35:30.085759 1148 raft_consensus.cc:1261] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Refusing update from remote peer f531c50d23964a129cb7c5ce591b1335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:30.085745 1579 raft_consensus.cc:1261] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb [term 1 FOLLOWER]: Refusing update from remote peer f531c50d23964a129cb7c5ce591b1335: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:35:30.086779 1697 consensus_queue.cc:1035] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:30.087090 1688 consensus_queue.cc:1035] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:35:30.097851 1700 mvcc.cc:205] Tried to move back new op lower bound from 7316981883221016576 to 7316981883120902144. Current Snapshot: MvccSnapshot[applied={T|T < 7316981883221016576}]
I20260810 14:35:30.112214 938 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:30.112512 938 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:35:30.115965 938 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Sending DeleteTablet for 3 replicas of tablet d5c411f7cd3a491dbeceb409a9c7908a
I20260810 14:35:30.116990 1546 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c411f7cd3a491dbeceb409a9c7908a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:30 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:45432
I20260810 14:35:30.117086 1120 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c411f7cd3a491dbeceb409a9c7908a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:30 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:40226
I20260810 14:35:30.117090 1349 tablet_service.cc:1514] Processing DeleteTablet for tablet d5c411f7cd3a491dbeceb409a9c7908a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:35:30 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:40624
I20260810 14:35:30.118155 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 996
I20260810 14:35:30.127213 1715 tablet_replica.cc:330] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335: stopping tablet replica
I20260810 14:35:30.127812 1715 raft_consensus.cc:2229] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:35:30.128594 1715 raft_consensus.cc:2258] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:30.131178 1713 tablet_replica.cc:330] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb: stopping tablet replica
I20260810 14:35:30.131703 1714 tablet_replica.cc:330] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51: stopping tablet replica
I20260810 14:35:30.132107 1714 raft_consensus.cc:2229] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:30.132624 1714 raft_consensus.cc:2258] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:30.133190 1713 raft_consensus.cc:2229] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:35:30.133641 1713 raft_consensus.cc:2258] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:35:30.134315 1715 ts_tablet_manager.cc:1905] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:30.139007 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.119151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.6.46.193:38933)
0810 14:35:30.119716 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.119722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.119741 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:35:30.119794 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.119800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.119880 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.119969 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:30.121941 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.123138 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.125112 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.125754 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.125873 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.131811 (+ 5938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:30.131824 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:30.131841 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:30.131878 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:30.135643 (+ 3765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:30.136479 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:30.136487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:30.136491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:30.136649 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:30.137085 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:30.137108 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:30.137112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:30.137592 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:30.137879 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.138430 (+ 551us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.138644 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445}
I20260810 14:35:30.139695 1216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:35:30.143042 1713 ts_tablet_manager.cc:1905] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:35:30.147004 1715 ts_tablet_manager.cc:1918] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:30.147188 1715 log.cc:1196] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/d5c411f7cd3a491dbeceb409a9c7908a
I20260810 14:35:30.147789 1715 ts_tablet_manager.cc:1939] T d5c411f7cd3a491dbeceb409a9c7908a P f531c50d23964a129cb7c5ce591b1335: Deleting consensus metadata
I20260810 14:35:30.149701 916 catalog_manager.cc:4841] TS f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835): tablet d5c411f7cd3a491dbeceb409a9c7908a (table test-table [id=fa4fbb8c625e4e19b5cf4dfbfe8425f8]) successfully deleted
I20260810 14:35:30.150471 1713 ts_tablet_manager.cc:1918] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:35:30.150629 1713 log.cc:1196] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/d5c411f7cd3a491dbeceb409a9c7908a
I20260810 14:35:30.151151 1713 ts_tablet_manager.cc:1939] T d5c411f7cd3a491dbeceb409a9c7908a P 07cc4f5ef6fa4877a026847e535b87eb: Deleting consensus metadata
I20260810 14:35:30.152827 918 catalog_manager.cc:4841] TS 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925): tablet d5c411f7cd3a491dbeceb409a9c7908a (table test-table [id=fa4fbb8c625e4e19b5cf4dfbfe8425f8]) successfully deleted
I20260810 14:35:30.231798 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 996
I20260810 14:35:30.261777 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 1235
W20260810 14:35:30.261793 917 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.46.193:38933 (error 108)
I20260810 14:35:30.263875 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.263442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:38933 (local address 127.0.0.1:40242)
0810 14:35:30.263620 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.263714 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:38933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 14:35:30.264101 917 catalog_manager.cc:4584] TS abb955bb82774b849f4e8f922af12e51 (127.6.46.193:38933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d5c411f7cd3a491dbeceb409a9c7908a: Network error: Client connection negotiation failed: client connection to 127.6.46.193:38933: connect: Connection refused (error 111)
I20260810 14:35:30.282369 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.262838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.6.46.194:45835)
0810 14:35:30.263043 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.263050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.263285 (+ 235us) server_negotiation.cc:407] Connection header received
0810 14:35:30.263601 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.263607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.263697 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.263796 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:30.265625 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.266886 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.268268 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.268951 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.269081 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.275640 (+ 6559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:30.275656 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:30.275675 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:30.275743 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:30.279541 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:30.280226 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:30.280235 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:30.280239 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:30.280412 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:30.280745 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:30.280750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:30.280754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:30.281197 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:30.281547 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.281699 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.281920 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 14:35:30.283191 1437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:35:30.317840 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.317274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:38933 (local address 127.0.0.1:40248)
0810 14:35:30.317504 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.317634 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:38933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 14:35:30.319100 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.318567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:38933 (local address 127.0.0.1:40258)
0810 14:35:30.318751 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.318894 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:38933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 14:35:30.379854 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.379248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:38933 (local address 127.0.0.1:40260)
0810 14:35:30.379510 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.379626 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:38933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 14:35:30.380909 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.380534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:38933 (local address 127.0.0.1:40262)
0810 14:35:30.380686 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.380774 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:38933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 14:35:30.397171 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 1235
I20260810 14:35:30.420150 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 1450
I20260810 14:35:30.433511 1670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.420997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.6.46.195:42925)
0810 14:35:30.421201 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:30.421206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:30.421225 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:35:30.421345 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:30.421349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:30.421417 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:30.421498 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:30.422854 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.423684 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.424960 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:30.425386 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:30.425495 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:30.428241 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:30.428256 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:30.428273 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:30.428315 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:30.431109 (+ 2794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:30.431766 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:30.431773 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:30.431775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:30.431918 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:30.432235 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:30.432239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:30.432241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:30.432656 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:30.432882 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:30.433088 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 14:35:30.433259 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 14:35:30.434185 1647 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:35:30.461344 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.460804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:38933 (local address 127.0.0.1:40278)
0810 14:35:30.461046 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.461165 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:38933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 14:35:30.462466 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.462035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:38933 (local address 127.0.0.1:40292)
0810 14:35:30.462244 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.462333 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:38933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260810 14:35:30.522961 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 1450
I20260810 14:35:30.550318 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 888
I20260810 14:35:30.551389 981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:35:30.666749 1671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:30.666095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:38933 (local address 127.0.0.1:40296)
0810 14:35:30.666435 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:30.666530 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:38933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194}
I20260810 14:35:30.666775 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 888
I20260810 14:35:50.694929 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:41101
--webserver_interface=127.6.46.254
--webserver_port=35593
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:41101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:35:51.028688 1724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:51.028956 1724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:51.029095 1724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:51.035987 1724 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 14:35:51.036083 1724 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 14:35:51.036147 1724 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 14:35:51.036216 1724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:35:51.036290 1724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:51.036384 1724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:51.036443 1724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:35:51.036516 1724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:35:51.045703 1724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:41101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:41101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=35593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:35:51.046861 1724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:51.048081 1724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:51.059064 1733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:51.059109 1742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:51.059149 1737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:51.059870 1724 server_base.cc:1034] running on GCE node
I20260810 14:35:51.060526 1724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:51.061704 1724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:51.062922 1724 hybrid_clock.cc:648] HybridClock initialized: now 1786372551062867 us; error 34 us; skew 500 ppm
I20260810 14:35:51.063220 1724 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:35:51 dist-test-slave-q9h9 krb5kdc[32420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:51.066182 1724 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:35:51.067498 1724 webserver.cc:460] Webserver started at http://127.6.46.254:35593/ using document root <none> and password file <none>
I20260810 14:35:51.068099 1724 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:51.068182 1724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:51.073119 1724 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 14:35:51.075829 1756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:51.076996 1724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:35:51.077139 1724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "4aab9c7489ad4f0fb41f5c240aca6b04"
format_stamp: "Formatted at 2026-08-10 14:35:05 on dist-test-slave-q9h9"
server_key: "d429999aa21d50299bd329d770bed149"
server_key_iv: "27b2ade9c96ef4de34a3ad4af1ee3bee"
server_key_version: "encryptionkey@0"
I20260810 14:35:51.077992 1724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:51.096338 1724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:51.099953 1724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:51.100181 1724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:51.117365 1724 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:41101
I20260810 14:35:51.117400 1832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:41101 every 8 connection(s)
I20260810 14:35:51.118732 1724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:35:51.124024 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 1724
I20260810 14:35:51.124837 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:38933
--local_ip_for_outbound_sockets=127.6.46.193
--tserver_master_addrs=127.6.46.254:41101
--webserver_port=41173
--webserver_interface=127.6.46.193
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:35:51.125269 1833 sys_catalog.cc:263] Verifying existing consensus state
I20260810 14:35:51.128367 1833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Bootstrap starting.
I20260810 14:35:51.148130 1833 log.cc:824] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:51.163622 1833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:35:51.164124 1833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Bootstrap complete.
I20260810 14:35:51.172657 1833 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } }
I20260810 14:35:51.173489 1833 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4aab9c7489ad4f0fb41f5c240aca6b04, State: Initialized, Role: FOLLOWER
I20260810 14:35:51.174127 1833 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } }
I20260810 14:35:51.174348 1833 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:35:51.174451 1833 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:35:51.174606 1833 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 2 FOLLOWER]: Advancing to term 3
I20260810 14:35:51.178319 1833 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } }
I20260810 14:35:51.178761 1833 leader_election.cc:302] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4aab9c7489ad4f0fb41f5c240aca6b04; no voters:
I20260810 14:35:51.179375 1833 leader_election.cc:288] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 14:35:51.179512 1840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 14:35:51.180562 1840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [term 3 LEADER]: Becoming Leader. State: Replica: 4aab9c7489ad4f0fb41f5c240aca6b04, State: Running, Role: LEADER
I20260810 14:35:51.181213 1840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } }
I20260810 14:35:51.181429 1833 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:35:51.187355 1842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } } }
I20260810 14:35:51.187945 1842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:51.187469 1843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4aab9c7489ad4f0fb41f5c240aca6b04. Latest consensus state: current_term: 3 leader_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4aab9c7489ad4f0fb41f5c240aca6b04" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 41101 } } }
I20260810 14:35:51.188429 1843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04 [sys.catalog]: This master's current role is: LEADER
I20260810 14:35:51.190857 1853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:35:51.204948 1853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:35:51.206590 1853 catalog_manager.cc:1259] Loaded cluster ID: 11c56c5a0c894737b87adfe1736be1ba
I20260810 14:35:51.206745 1853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:35:51.211483 1853 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:35:51.212898 1853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Loaded TSK: 0
I20260810 14:35:51.213009 1853 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4aab9c7489ad4f0fb41f5c240aca6b04: Loaded TSK: 1
F20260810 14:35:51.214262 1853 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f990802a9ad google::LogMessage::Fail() at ??:0
@ 0x7f990802d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f990802a4de google::LogMessage::Flush() at ??:0
@ 0x7f990802ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f990bd8e7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f990bd7d73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f991da591c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f991da58783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f991dae42a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f991dae40c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f991db2e5e9 std::function<>::operator()() at ??:0
@ 0x7f991da5ccdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f991da5ac36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f991dae2610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f991dae2439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f990ac76345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f990ac7991c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f990ac79799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f990ac54eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f990df2e6db start_thread at ??:0
@ 0x7f9906ec461f clone at ??:0
W20260810 14:35:51.489133 1836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:51.489404 1836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:51.489550 1836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:51.497002 1836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:51.497129 1836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:51.497299 1836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:35:51.506642 1836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:38933
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=41173
--tserver_master_addrs=127.6.46.254:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:35:51.507808 1836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:51.509027 1836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:51.521420 1909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:51.521466 1902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:51.521499 1911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:51.522526 1836 server_base.cc:1034] running on GCE node
I20260810 14:35:51.523089 1836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:51.523974 1836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:51.525220 1836 hybrid_clock.cc:648] HybridClock initialized: now 1786372551525156 us; error 32 us; skew 500 ppm
I20260810 14:35:51.525535 1836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:35:51 dist-test-slave-q9h9 krb5kdc[32420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:51.528594 1836 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:35:51.530112 1836 webserver.cc:460] Webserver started at http://127.6.46.193:41173/ using document root <none> and password file <none>
I20260810 14:35:51.530762 1836 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:51.530879 1836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:51.536082 1836 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 14:35:51.538894 1923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:51.540128 1836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:35:51.540287 1836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "abb955bb82774b849f4e8f922af12e51"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "6b814150aa85b6ddc6ecde301692321a"
server_key_iv: "64fe1906d0e1a87825e10293bc24c67e"
server_key_version: "encryptionkey@0"
I20260810 14:35:51.541206 1836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:51.559343 1836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:51.563251 1836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:51.563505 1836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:51.564447 1836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:51.566207 1935 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260810 14:35:51.571774 1836 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 14:35:51.571882 1836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260810 14:35:51.571996 1836 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 14:35:51.575189 1836 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 14:35:51.575263 1836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20260810 14:35:51.575584 1935 tablet_bootstrap.cc:492] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51: Bootstrap starting.
I20260810 14:35:51.583879 1935 log.cc:824] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51: Log is configured to *not* fsync() on all Append() calls
I20260810 14:35:51.602653 1935 tablet_bootstrap.cc:492] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 14:35:51.603557 1935 tablet_bootstrap.cc:492] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51: Bootstrap complete.
I20260810 14:35:51.604130 1935 ts_tablet_manager.cc:1397] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.005s
I20260810 14:35:51.613457 1935 raft_consensus.cc:348] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:51.614620 1935 raft_consensus.cc:729] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb955bb82774b849f4e8f922af12e51, State: Initialized, Role: FOLLOWER
I20260810 14:35:51.615232 1935 consensus_queue.cc:260] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:51.616672 1935 ts_tablet_manager.cc:1428] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260810 14:35:51.618680 1836 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:38933
I20260810 14:35:51.618731 2116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:38933 every 8 connection(s)
I20260810 14:35:51.619778 1836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:35:51.622273 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 1836
I20260810 14:35:51.622996 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:45835
--local_ip_for_outbound_sockets=127.6.46.194
--tserver_master_addrs=127.6.46.254:41101
--webserver_port=35991
--webserver_interface=127.6.46.194
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:35:51.624395 2123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:51.622197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:59371)
0810 14:35:51.623070 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:51.623325 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":359,"threads_started":1}
W20260810 14:35:51.625069 2117 heartbeater.cc:644] Failed to heartbeat to 127.6.46.254:41101 (0 consecutive failures): Network error: Failed to ping master at 127.6.46.254:41101: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
I20260810 14:35:51.626361 2123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:51.625572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:46273)
0810 14:35:51.626152 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:51.626230 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483}
I20260810 14:35:51.627374 2123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:51.626961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:39647)
0810 14:35:51.627136 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:51.627230 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 14:35:51.627674 2117 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:35:51.947021 2127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:51.947273 2127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:51.947371 2127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:51.954229 2127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:51.954321 2127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:51.954433 2127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:35:51.963516 2127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:45835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=35991
--tserver_master_addrs=127.6.46.254:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:35:51.964609 2127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:51.965792 2127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:51.978053 2133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:51.978138 2136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:51.978107 2138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:51.978892 2127 server_base.cc:1034] running on GCE node
I20260810 14:35:51.979420 2127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:51.980234 2127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:51.981421 2127 hybrid_clock.cc:648] HybridClock initialized: now 1786372551981381 us; error 31 us; skew 500 ppm
I20260810 14:35:51.981714 2127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:35:51 dist-test-slave-q9h9 krb5kdc[32420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:51.984678 2127 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:35:51.986069 2127 webserver.cc:460] Webserver started at http://127.6.46.194:35991/ using document root <none> and password file <none>
I20260810 14:35:51.986699 2127 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:51.986812 2127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:51.991796 2127 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 14:35:51.994395 2149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:51.995610 2127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:51.995752 2127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "f531c50d23964a129cb7c5ce591b1335"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "831ab8b045f379a9de6dde5d0c721f96"
server_key_iv: "619efde213a57ffd10a2fe6898d502e0"
server_key_version: "encryptionkey@0"
I20260810 14:35:51.996685 2127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:52.018787 2127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:52.022825 2127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:52.023078 2127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:52.024016 2127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:52.025306 2127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:52.025403 2127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:52.025509 2127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:52.025586 2127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:52.055898 2127 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:45835
I20260810 14:35:52.055962 2343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:45835 every 8 connection(s)
I20260810 14:35:52.057286 2127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:35:52.061847 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:52.060036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:42783)
0810 14:35:52.060751 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:52.060978 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":276,"threads_started":1}
I20260810 14:35:52.061924 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 2127
W20260810 14:35:52.062542 2345 heartbeater.cc:644] Failed to heartbeat to 127.6.46.254:41101 (0 consecutive failures): Network error: Failed to ping master at 127.6.46.254:41101: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
I20260810 14:35:52.062564 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:42925
--local_ip_for_outbound_sockets=127.6.46.195
--tserver_master_addrs=127.6.46.254:41101
--webserver_port=34151
--webserver_interface=127.6.46.195
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:41775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:35:52.063334 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:52.063022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:37777)
0810 14:35:52.063167 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:52.063233 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 14:35:52.064121 2348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:52.063805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:58719)
0810 14:35:52.063945 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:52.064017 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 14:35:52.064352 2345 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 14:35:52.383287 2351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:35:52.383535 2351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:35:52.383670 2351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:35:52.390599 2351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:35:52.390688 2351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:35:52.390827 2351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:35:52.399957 2351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:41775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:42925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=34151
--tserver_master_addrs=127.6.46.254:41101
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:35:52.401139 2351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:35:52.402369 2351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:35:52.414415 2364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:52.414371 2366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:35:52.414777 2368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:35:52.415307 2351 server_base.cc:1034] running on GCE node
I20260810 14:35:52.415835 2351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:35:52.416662 2351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:35:52.417851 2351 hybrid_clock.cc:648] HybridClock initialized: now 1786372552417805 us; error 38 us; skew 500 ppm
I20260810 14:35:52.418172 2351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:35:52 dist-test-slave-q9h9 krb5kdc[32420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372552, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:35:52.421065 2351 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:35:52.422410 2351 webserver.cc:460] Webserver started at http://127.6.46.195:34151/ using document root <none> and password file <none>
I20260810 14:35:52.423012 2351 fs_manager.cc:359] Metadata directory not provided
I20260810 14:35:52.423132 2351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:35:52.428047 2351 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 14:35:52.430565 2379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:52.431841 2351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:35:52.432022 2351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "07cc4f5ef6fa4877a026847e535b87eb"
format_stamp: "Formatted at 2026-08-10 14:35:06 on dist-test-slave-q9h9"
server_key: "6804b4ef75e56e88264d941889208a2e"
server_key_iv: "7dae886e82cdc3d0350892af32034d39"
server_key_version: "encryptionkey@0"
I20260810 14:35:52.432971 2351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:35:52.444850 2351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:35:52.448734 2351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:35:52.448971 2351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:35:52.449838 2351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:35:52.451088 2351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:35:52.451192 2351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:52.451293 2351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:35:52.451363 2351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:35:52.481276 2351 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:42925
I20260810 14:35:52.481333 2572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:42925 every 8 connection(s)
I20260810 14:35:52.482626 2351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:35:52.487031 2576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:52.485288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:47845)
0810 14:35:52.485964 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:52.486209 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":263,"threads_started":1}
W20260810 14:35:52.487691 2574 heartbeater.cc:644] Failed to heartbeat to 127.6.46.254:41101 (0 consecutive failures): Network error: Failed to ping master at 127.6.46.254:41101: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
I20260810 14:35:52.488487 2576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:52.488120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:54871)
0810 14:35:52.488271 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:52.488354 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 14:35:52.488669 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 2351
I20260810 14:35:52.489375 2576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:52.489041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:35895)
0810 14:35:52.489179 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:52.489270 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 14:35:52.489588 2574 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 14:35:52.628721 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:52.628070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:40661)
0810 14:35:52.628465 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:52.628547 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":189,"threads_started":1}
I20260810 14:35:52.791044 2579 raft_consensus.cc:482] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:52.791319 2579 raft_consensus.cc:504] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:52.793402 2579 leader_election.cc:288] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925), f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835)
Aug 10 14:35:52 dist-test-slave-q9h9 krb5kdc[32420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
Aug 10 14:35:52 dist-test-slave-q9h9 krb5kdc[32420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372551, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:35:52.817342 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:52.793190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:45835 (local address 127.6.46.193:49243)
0810 14:35:52.793820 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:52.793867 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:52.794572 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:52.798335 (+ 3763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:52.798347 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:52.799344 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:52.800468 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:52.800496 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:52.802805 (+ 2309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:52.802810 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:52.805127 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:52.805150 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:52.806471 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:52.806478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:52.806843 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:52.807672 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:52.807703 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:52.810554 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:52.813769 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:52.813778 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:52.813791 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:52.814465 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:52.815018 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:52.815025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:52.815027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:52.815184 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:52.815899 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:52.815907 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:52.816195 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 14:35:52.816430 (+ 235us) client_negotiation.cc:240] Negotiation successful
0810 14:35:52.816913 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":341,"spinlock_wait_cycles":65280}
I20260810 14:35:52.817432 2584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:52.793581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:54349 (local address 127.6.46.195:42925)
0810 14:35:52.795039 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:52.795053 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:52.795103 (+ 50us) server_negotiation.cc:407] Connection header received
0810 14:35:52.797314 (+ 2211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:52.797327 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:52.797796 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:52.798187 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:52.800669 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:52.802605 (+ 1936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:52.805279 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:52.806287 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:52.806475 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:52.810988 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:52.810995 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:52.811006 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:52.811040 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:52.813707 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:52.814598 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:52.814607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:52.814614 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:52.814836 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:52.815284 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:52.815289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:52.815293 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:52.815774 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:52.815941 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:52.816699 (+ 758us) server_negotiation.cc:299] Negotiation successful
0810 14:35:52.817069 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1281,"thread_start_us":1116,"threads_started":1}
I20260810 14:35:52.817502 2583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:52.793662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:49243 (local address 127.6.46.194:45835)
0810 14:35:52.795295 (+ 1633us) server_negotiation.cc:206] Beginning negotiation
0810 14:35:52.795307 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:35:52.795340 (+ 33us) server_negotiation.cc:407] Connection header received
0810 14:35:52.797315 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:35:52.797327 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:35:52.797796 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:35:52.798187 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:35:52.800677 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:52.802605 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:52.805279 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:35:52.806287 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:35:52.806477 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:52.810839 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:35:52.810850 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:35:52.810867 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:35:52.810914 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:35:52.813616 (+ 2702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:52.814678 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:52.814686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:52.814693 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:52.814900 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:35:52.815300 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:35:52.815305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:35:52.815307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:35:52.815752 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:35:52.815907 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 14:35:52.816706 (+ 799us) server_negotiation.cc:299] Negotiation successful
0810 14:35:52.817079 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1471,"thread_start_us":1362,"threads_started":1}
I20260810 14:35:52.817487 2580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:52.793277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:42925 (local address 127.6.46.193:54349)
0810 14:35:52.794049 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:52.794069 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:35:52.794926 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:35:52.798335 (+ 3409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:35:52.798347 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:35:52.799344 (+ 997us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:35:52.800468 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:52.800496 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:52.802801 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:52.802807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:52.805127 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:35:52.805143 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:35:52.806507 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:35:52.806515 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:35:52.806845 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:35:52.807686 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:35:52.807713 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:35:52.810743 (+ 3030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:35:52.813843 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:52.813849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:52.813853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:52.814452 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:52.814964 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:35:52.814969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:35:52.814972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:35:52.815166 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:35:52.815910 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:35:52.815914 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:35:52.816185 (+ 271us) client_negotiation.cc:769] Sending connection context
0810 14:35:52.816430 (+ 245us) client_negotiation.cc:240] Negotiation successful
0810 14:35:52.817072 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":551,"spinlock_wait_cycles":54912,"thread_start_us":491,"threads_started":1}
W20260810 14:35:52.820966 1924 leader_election.cc:341] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835): Not found: Tablet not found: d5c411f7cd3a491dbeceb409a9c7908a
W20260810 14:35:52.821269 1926 leader_election.cc:341] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925): Not found: Tablet not found: d5c411f7cd3a491dbeceb409a9c7908a
I20260810 14:35:52.821385 1926 leader_election.cc:302] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abb955bb82774b849f4e8f922af12e51; no voters: 07cc4f5ef6fa4877a026847e535b87eb, f531c50d23964a129cb7c5ce591b1335
I20260810 14:35:52.821725 2579 raft_consensus.cc:2726] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:35:53.065631 2586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.064801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:41149)
0810 14:35:53.065256 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:53.065392 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":220,"threads_started":1}
I20260810 14:35:53.490739 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.490053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:52209)
0810 14:35:53.490453 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:53.490552 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":192,"threads_started":1}
I20260810 14:35:53.630328 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:53.629574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:51303)
0810 14:35:53.630085 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:53.630174 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":270,"threads_started":1}
I20260810 14:35:54.067067 2592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.066428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:34691)
0810 14:35:54.066849 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.066924 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":231,"threads_started":1}
I20260810 14:35:54.492115 2593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.491443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:54721)
0810 14:35:54.491872 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.491955 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":221,"threads_started":1}
I20260810 14:35:54.631841 2594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:54.631126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:47979)
0810 14:35:54.631587 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:54.631667 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":215,"threads_started":1}
I20260810 14:35:54.967689 2595 raft_consensus.cc:482] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:54.967882 2595 raft_consensus.cc:504] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:54.968533 2595 leader_election.cc:288] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925), f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835)
W20260810 14:35:54.969523 1924 leader_election.cc:341] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835): Not found: Tablet not found: d5c411f7cd3a491dbeceb409a9c7908a
W20260810 14:35:54.969743 1926 leader_election.cc:341] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925): Not found: Tablet not found: d5c411f7cd3a491dbeceb409a9c7908a
I20260810 14:35:54.969872 1926 leader_election.cc:302] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abb955bb82774b849f4e8f922af12e51; no voters: 07cc4f5ef6fa4877a026847e535b87eb, f531c50d23964a129cb7c5ce591b1335
I20260810 14:35:54.970144 2595 raft_consensus.cc:2726] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:35:55.068471 2597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:55.067825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:49429)
0810 14:35:55.068242 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:55.068317 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":224,"threads_started":1}
I20260810 14:35:55.493687 2600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:55.492926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:36163)
0810 14:35:55.493442 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:55.493527 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":308,"threads_started":1}
I20260810 14:35:55.633530 2601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:55.632700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:55253)
0810 14:35:55.633212 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:55.633313 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":273,"threads_started":1}
I20260810 14:35:56.069897 2602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.069237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:39851)
0810 14:35:56.069646 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.069723 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":192,"threads_started":1}
I20260810 14:35:56.495175 2603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.494473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:37743)
0810 14:35:56.494905 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.494989 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":204,"threads_started":1}
I20260810 14:35:56.635478 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:56.634727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:48185)
0810 14:35:56.635213 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:56.635299 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":251,"threads_started":1}
I20260810 14:35:56.721879 2607 raft_consensus.cc:482] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:35:56.722110 2607 raft_consensus.cc:504] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:35:56.722774 2607 leader_election.cc:288] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925), f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835)
W20260810 14:35:56.723713 1924 leader_election.cc:341] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835): Not found: Tablet not found: d5c411f7cd3a491dbeceb409a9c7908a
W20260810 14:35:56.723889 1926 leader_election.cc:341] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925): Not found: Tablet not found: d5c411f7cd3a491dbeceb409a9c7908a
I20260810 14:35:56.724013 1926 leader_election.cc:302] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abb955bb82774b849f4e8f922af12e51; no voters: 07cc4f5ef6fa4877a026847e535b87eb, f531c50d23964a129cb7c5ce591b1335
I20260810 14:35:56.724246 2607 raft_consensus.cc:2726] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:35:57.071522 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:57.070744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:49721)
0810 14:35:57.071235 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:57.071336 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":285,"threads_started":1}
I20260810 14:35:57.496615 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:57.495952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:37473)
0810 14:35:57.496380 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:57.496466 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":233,"threads_started":1}
I20260810 14:35:57.636880 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:57.636228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:44631)
0810 14:35:57.636635 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:57.636721 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":203,"threads_started":1}
I20260810 14:35:58.073002 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:58.072277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:36557)
0810 14:35:58.072750 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:58.072828 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":267,"threads_started":1}
I20260810 14:35:58.498129 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:58.497398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:37927)
0810 14:35:58.497844 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:58.497921 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":233,"threads_started":1}
I20260810 14:35:58.638362 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:58.637682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:47195)
0810 14:35:58.638102 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:58.638176 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":204,"threads_started":1}
I20260810 14:35:59.074712 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.073815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:36399)
0810 14:35:59.074425 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.074517 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":281,"threads_started":1}
I20260810 14:35:59.499769 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.498943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:41707)
0810 14:35:59.499471 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.499553 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":236,"threads_started":1}
I20260810 14:35:59.639778 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:35:59.639122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:39013)
0810 14:35:59.639537 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0810 14:35:59.639623 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":199,"threads_started":1}
I20260810 14:36:00.076161 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:00.075525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:35631)
0810 14:36:00.075909 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:00.075985 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":196,"threads_started":1}
I20260810 14:36:00.133188 2627 raft_consensus.cc:482] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:36:00.133399 2627 raft_consensus.cc:504] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:36:00.134100 2627 leader_election.cc:288] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925), f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835)
W20260810 14:36:00.135025 1924 leader_election.cc:341] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835): Not found: Tablet not found: d5c411f7cd3a491dbeceb409a9c7908a
W20260810 14:36:00.135205 1926 leader_election.cc:341] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925): Not found: Tablet not found: d5c411f7cd3a491dbeceb409a9c7908a
I20260810 14:36:00.135327 1926 leader_election.cc:302] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abb955bb82774b849f4e8f922af12e51; no voters: 07cc4f5ef6fa4877a026847e535b87eb, f531c50d23964a129cb7c5ce591b1335
I20260810 14:36:00.135555 2627 raft_consensus.cc:2726] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:36:00.501332 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:00.500613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:55191)
0810 14:36:00.501035 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:00.501142 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":202,"threads_started":1}
I20260810 14:36:00.641242 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:00.640566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:42587)
0810 14:36:00.640967 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:00.641072 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":200,"threads_started":1}
I20260810 14:36:01.077565 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:01.076897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:45327)
0810 14:36:01.077339 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:01.077425 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":206,"threads_started":1}
I20260810 14:36:01.502871 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:01.502196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:54143)
0810 14:36:01.502639 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:01.502725 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":206,"threads_started":1}
I20260810 14:36:01.642951 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:01.642227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:39749)
0810 14:36:01.642657 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:01.642744 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":220,"threads_started":1}
I20260810 14:36:02.079048 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:02.078378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:58471)
0810 14:36:02.078784 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:02.078860 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":217,"threads_started":1}
I20260810 14:36:02.504360 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:02.503716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:43599)
0810 14:36:02.504119 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:02.504195 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":214,"threads_started":1}
I20260810 14:36:02.644399 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:02.643757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:45507)
0810 14:36:02.644151 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:02.644235 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":204,"threads_started":1}
I20260810 14:36:03.080538 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:03.079841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:37855)
0810 14:36:03.080301 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:03.080383 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":225,"threads_started":1}
I20260810 14:36:03.505829 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:03.505127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:40869)
0810 14:36:03.505593 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:03.505674 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":221,"threads_started":1}
I20260810 14:36:03.645804 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:03.645125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:47707)
0810 14:36:03.645556 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:03.645648 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":265,"threads_started":1}
I20260810 14:36:04.082119 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:04.081377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:37489)
0810 14:36:04.081812 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:04.081897 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":218,"threads_started":1}
I20260810 14:36:04.507300 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:04.506605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:52699)
0810 14:36:04.507058 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:04.507137 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":238,"threads_started":1}
I20260810 14:36:04.647305 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:04.646595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:55807)
0810 14:36:04.647041 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:04.647130 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":235,"threads_started":1}
I20260810 14:36:05.083611 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:05.082924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:57251)
0810 14:36:05.083357 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:05.083450 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":220,"threads_started":1}
I20260810 14:36:05.508845 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:05.508143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:35725)
0810 14:36:05.508586 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:05.508685 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":214,"threads_started":1}
I20260810 14:36:05.648942 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:05.648118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:42183)
0810 14:36:05.648682 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:05.648764 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":331,"threads_started":1}
I20260810 14:36:06.085109 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:06.084407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:43341)
0810 14:36:06.084841 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:06.084917 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":243,"threads_started":1}
I20260810 14:36:06.510387 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:06.509634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:58357)
0810 14:36:06.510099 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:06.510188 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":222,"threads_started":1}
I20260810 14:36:06.650707 2656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:06.649954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.193:45393)
0810 14:36:06.650422 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:06.650512 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":244,"threads_started":1}
I20260810 14:36:06.967072 2657 raft_consensus.cc:482] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:36:06.967278 2657 raft_consensus.cc:504] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07cc4f5ef6fa4877a026847e535b87eb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 42925 } } peers { permanent_uuid: "abb955bb82774b849f4e8f922af12e51" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 38933 } } peers { permanent_uuid: "f531c50d23964a129cb7c5ce591b1335" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45835 } }
I20260810 14:36:06.967981 2657 leader_election.cc:288] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925), f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835)
W20260810 14:36:06.969018 1926 leader_election.cc:341] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 07cc4f5ef6fa4877a026847e535b87eb (127.6.46.195:42925): Not found: Tablet not found: d5c411f7cd3a491dbeceb409a9c7908a
W20260810 14:36:06.969223 1924 leader_election.cc:341] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f531c50d23964a129cb7c5ce591b1335 (127.6.46.194:45835): Not found: Tablet not found: d5c411f7cd3a491dbeceb409a9c7908a
I20260810 14:36:06.969349 1924 leader_election.cc:302] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: abb955bb82774b849f4e8f922af12e51; no voters: 07cc4f5ef6fa4877a026847e535b87eb, f531c50d23964a129cb7c5ce591b1335
I20260810 14:36:06.969595 2657 raft_consensus.cc:2726] T d5c411f7cd3a491dbeceb409a9c7908a P abb955bb82774b849f4e8f922af12e51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 14:36:07.086923 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:07.086215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.194:34357)
0810 14:36:07.086643 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:07.086760 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":211,"threads_started":1}
I20260810 14:36:07.491031 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 1836
I20260810 14:36:07.505332 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:07.492033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32972 (local address 127.6.46.193:38933)
0810 14:36:07.492571 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:07.492578 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:07.492603 (+ 25us) server_negotiation.cc:407] Connection header received
0810 14:36:07.492663 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:07.492672 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:07.492980 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:07.493159 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:07.494935 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.495851 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.497030 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.497566 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.497658 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:07.500265 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:07.500275 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:07.500289 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:07.500323 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:07.502980 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:07.503609 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:07.503615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:07.503617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:07.503747 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:07.504019 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:07.504022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:07.504024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:07.504354 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:07.504578 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:07.504804 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 14:36:07.504993 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":249,"threads_started":1}
I20260810 14:36:07.506554 2108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:07.512023 2665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:07.511304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:41101 (local address 127.6.46.195:57535)
0810 14:36:07.511774 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:07.511887 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.254:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":237,"threads_started":1}
I20260810 14:36:07.592728 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 1836
I20260810 14:36:07.615520 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 2127
I20260810 14:36:07.628752 2666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:07.616253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44680 (local address 127.6.46.194:45835)
0810 14:36:07.616694 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:07.616698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:07.616715 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:36:07.616770 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:07.616774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:07.616835 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:07.616933 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:07.618299 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.619144 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.620320 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.620727 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.620818 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:07.623404 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:07.623409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:07.623412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:07.623445 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:07.625826 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:07.626610 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:07.626615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:07.626619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:07.626788 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:07.627164 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:07.627168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:07.627171 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:07.627487 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:07.627744 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:07.628080 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 14:36:07.628331 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":232,"threads_started":1}
I20260810 14:36:07.629668 2338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:07.714197 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 2127
I20260810 14:36:07.738107 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 2351
I20260810 14:36:07.751077 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:07.738902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42140 (local address 127.6.46.195:42925)
0810 14:36:07.739428 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:07.739433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:07.739448 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:36:07.739504 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:07.739508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:07.739579 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:07.739679 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:07.741032 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.741882 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.743120 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:07.743586 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:07.743690 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:07.746177 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:07.746182 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:07.746185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:07.746216 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:07.748676 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:07.749264 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:07.749267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:07.749269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:07.749405 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:07.749729 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:07.749732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:07.749734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:07.750039 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:07.750275 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:07.750511 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 14:36:07.750709 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":261,"threads_started":1}
I20260810 14:36:07.751776 2564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:07.834102 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 2351
2026-08-10T14:36:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64241 ms)
[----------] 30 tests from SecurityITest (292605 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:07 dist-test-slave-q9h9 krb5kdc[2675](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:07 dist-test-slave-q9h9 krb5kdc[2675](info): set up 2 sockets
Aug 10 14:36:07 dist-test-slave-q9h9 krb5kdc[2675](info): commencing operation
krb5kdc: starting...
W20260810 14:36:09.610502 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.733s user 0.001s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:09 dist-test-slave-q9h9 krb5kdc[2675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:36:09.874166 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:37611
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:35293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:37611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:10.197302 2698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:10.197566 2698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:10.197692 2698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:10.204741 2698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:10.204855 2698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:10.204933 2698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:10.204998 2698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:10.205050 2698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:10.214514 2698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:35293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:37611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:37611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:10.215781 2698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:10.217013 2698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:10.228240 2707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:10.228192 2712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:10.228283 2710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:10.228446 2698 server_base.cc:1034] running on GCE node
I20260810 14:36:10.229375 2698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:10.230677 2698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:10.231877 2698 hybrid_clock.cc:648] HybridClock initialized: now 1786372570231835 us; error 27 us; skew 500 ppm
I20260810 14:36:10.232172 2698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:10 dist-test-slave-q9h9 krb5kdc[2675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372570, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:10.235143 2698 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:36:10.236604 2698 webserver.cc:460] Webserver started at http://127.6.46.254:46269/ using document root <none> and password file <none>
I20260810 14:36:10.237236 2698 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:10.237380 2698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:10.237713 2698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:10.240779 2698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "391bf123d6cc410e9e945f6c2a3fa028"
format_stamp: "Formatted at 2026-08-10 14:36:10 on dist-test-slave-q9h9"
server_key: "c3dc0d55b5b020563cbc91f40de30ebe"
server_key_iv: "d00eb32c79c7d02ece017e61154d6360"
server_key_version: "encryptionkey@0"
I20260810 14:36:10.241513 2698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "391bf123d6cc410e9e945f6c2a3fa028"
format_stamp: "Formatted at 2026-08-10 14:36:10 on dist-test-slave-q9h9"
server_key: "c3dc0d55b5b020563cbc91f40de30ebe"
server_key_iv: "d00eb32c79c7d02ece017e61154d6360"
server_key_version: "encryptionkey@0"
I20260810 14:36:10.246454 2698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 14:36:10.249637 2723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:10.250908 2698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:10.251050 2698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "391bf123d6cc410e9e945f6c2a3fa028"
format_stamp: "Formatted at 2026-08-10 14:36:10 on dist-test-slave-q9h9"
server_key: "c3dc0d55b5b020563cbc91f40de30ebe"
server_key_iv: "d00eb32c79c7d02ece017e61154d6360"
server_key_version: "encryptionkey@0"
I20260810 14:36:10.251221 2698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:10.282531 2698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:10.286175 2698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:10.286401 2698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:10.303645 2698 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:37611
I20260810 14:36:10.303709 2809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:37611 every 8 connection(s)
I20260810 14:36:10.304912 2698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:10.306620 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 2698
I20260810 14:36:10.306851 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:10.307689 6331 external_mini_cluster.cc:1428] Setting key e9f6277f9f9a0a7c1696bbde27c92494
I20260810 14:36:10.309432 2814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 14:36:10 dist-test-slave-q9h9 krb5kdc[2675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:36:10.322804 2814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028: Bootstrap starting.
I20260810 14:36:10.325843 2814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:10.326900 2814 log.cc:824] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:10.329587 2814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028: No bootstrap required, opened a new log
I20260810 14:36:10.330202 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:10.309787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.6.46.254:37611)
0810 14:36:10.310520 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:10.310531 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:10.310571 (+ 40us) server_negotiation.cc:407] Connection header received
0810 14:36:10.312126 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:10.312147 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:10.312508 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:10.312837 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:10.314647 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.316032 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.317794 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.318475 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.318615 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:10.323320 (+ 4705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:10.323327 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:10.323340 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:10.323374 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:10.326334 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:10.326984 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:10.326991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:10.326997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:10.327184 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:10.327547 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:10.327553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:10.327556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:10.328058 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:10.328329 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:10.328819 (+ 490us) server_negotiation.cc:299] Negotiation successful
0810 14:36:10.329204 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":325,"threads_started":1}
I20260810 14:36:10.337215 2814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391bf123d6cc410e9e945f6c2a3fa028" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 37611 } }
I20260810 14:36:10.337535 2814 raft_consensus.cc:374] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:10.337625 2814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391bf123d6cc410e9e945f6c2a3fa028, State: Initialized, Role: FOLLOWER
I20260810 14:36:10.338192 2814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391bf123d6cc410e9e945f6c2a3fa028" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 37611 } }
I20260810 14:36:10.338423 2814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:10.338548 2814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:10.338714 2814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:10.341045 2814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391bf123d6cc410e9e945f6c2a3fa028" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 37611 } }
I20260810 14:36:10.341501 2814 leader_election.cc:302] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 391bf123d6cc410e9e945f6c2a3fa028; no voters:
I20260810 14:36:10.342067 2814 leader_election.cc:288] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:10.342175 2820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:10.342829 2820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [term 1 LEADER]: Becoming Leader. State: Replica: 391bf123d6cc410e9e945f6c2a3fa028, State: Running, Role: LEADER
I20260810 14:36:10.343288 2820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391bf123d6cc410e9e945f6c2a3fa028" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 37611 } }
I20260810 14:36:10.344137 2814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:10.346130 2821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "391bf123d6cc410e9e945f6c2a3fa028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391bf123d6cc410e9e945f6c2a3fa028" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 37611 } } }
I20260810 14:36:10.346104 2822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391bf123d6cc410e9e945f6c2a3fa028. Latest consensus state: current_term: 1 leader_uuid: "391bf123d6cc410e9e945f6c2a3fa028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391bf123d6cc410e9e945f6c2a3fa028" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 37611 } } }
I20260810 14:36:10.346619 2821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:10.346621 2822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:10.347952 2827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:10.352949 2827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:10.362376 2827 catalog_manager.cc:1347] Generated new cluster ID: d6b67f8a05d740328d250db929960560
I20260810 14:36:10.362486 2827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:10.375288 2827 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:10.376534 2827 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:10.382093 2827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028: Generated new TSK 0
I20260810 14:36:10.382788 2827 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:36:10.450640 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:37611
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:35293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:10.771075 2849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:10.771327 2849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:10.771461 2849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:10.778326 2849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:10.778420 2849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:10.778563 2849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:36:10.787958 2849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:35293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:37611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:10.789095 2849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:10.790400 2849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:10.802711 2856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:10.802775 2855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:10.802752 2861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:10.803586 2849 server_base.cc:1034] running on GCE node
I20260810 14:36:10.804121 2849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:10.804975 2849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:10.806166 2849 hybrid_clock.cc:648] HybridClock initialized: now 1786372570806105 us; error 42 us; skew 500 ppm
I20260810 14:36:10.806432 2849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:10 dist-test-slave-q9h9 krb5kdc[2675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372570, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:10.809371 2849 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:36:10.810730 2849 webserver.cc:460] Webserver started at http://127.6.46.193:45449/ using document root <none> and password file <none>
I20260810 14:36:10.811353 2849 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:10.811467 2849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:10.811769 2849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:10.814618 2849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "08ca3ff693e64349b934f8f9727fe046"
format_stamp: "Formatted at 2026-08-10 14:36:10 on dist-test-slave-q9h9"
server_key: "91f48cbfb4451aaa5d7a24094ab62460"
server_key_iv: "286b798683f6f1132a2c55c557b8e5be"
server_key_version: "encryptionkey@0"
I20260810 14:36:10.815373 2849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "08ca3ff693e64349b934f8f9727fe046"
format_stamp: "Formatted at 2026-08-10 14:36:10 on dist-test-slave-q9h9"
server_key: "91f48cbfb4451aaa5d7a24094ab62460"
server_key_iv: "286b798683f6f1132a2c55c557b8e5be"
server_key_version: "encryptionkey@0"
I20260810 14:36:10.820199 2849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:36:10.823426 2872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:10.824636 2849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 14:36:10.824766 2849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "08ca3ff693e64349b934f8f9727fe046"
format_stamp: "Formatted at 2026-08-10 14:36:10 on dist-test-slave-q9h9"
server_key: "91f48cbfb4451aaa5d7a24094ab62460"
server_key_iv: "286b798683f6f1132a2c55c557b8e5be"
server_key_version: "encryptionkey@0"
I20260810 14:36:10.824919 2849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:10.843397 2849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:10.847338 2849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:10.847606 2849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:10.848492 2849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:10.849766 2849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:10.849845 2849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:10.849960 2849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:10.850037 2849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:10.879369 2849 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:44841
I20260810 14:36:10.879410 3069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:44841 every 8 connection(s)
I20260810 14:36:10.880538 2849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:36:10.884038 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 2849
I20260810 14:36:10.884267 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:36:10.885053 6331 external_mini_cluster.cc:1428] Setting key bbdea6959e6f308077500e23609c0e4a
Aug 10 14:36:10 dist-test-slave-q9h9 krb5kdc[2675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372570, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:36:10.900254 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:10.883191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:55063 (local address 127.6.46.254:37611)
0810 14:36:10.883594 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:10.883599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:10.884607 (+ 1008us) server_negotiation.cc:407] Connection header received
0810 14:36:10.886520 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:10.886524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:10.886588 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:10.886687 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:10.888118 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.889109 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.890582 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.891126 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.891244 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:10.894509 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:10.894516 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:10.894521 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:10.894557 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:10.897441 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:10.898101 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:10.898106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:10.898109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:10.898251 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:10.898620 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:10.898624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:10.898627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:10.898963 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:10.899100 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:10.899718 (+ 618us) server_negotiation.cc:299] Negotiation successful
0810 14:36:10.899926 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":211,"threads_started":1}
I20260810 14:36:10.901060 3075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:10.883687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:37611 (local address 127.6.46.193:55063)
0810 14:36:10.884473 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:10.884508 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:10.886252 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:10.886911 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:10.886918 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:10.887400 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:10.887948 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:10.887965 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.889236 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.889240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:10.890448 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:10.890460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:10.891301 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:10.891306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:10.891490 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:10.892096 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:10.892122 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:10.894305 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:10.897560 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:10.897567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:10.897577 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:10.897958 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:10.898340 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:10.898343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:10.898346 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:10.898515 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:10.899082 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:10.899088 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:10.899376 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 14:36:10.899582 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 14:36:10.899912 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":295,"threads_started":1}
I20260810 14:36:10.902488 3070 heartbeater.cc:347] Connected to a master server at 127.6.46.254:37611
I20260810 14:36:10.902760 3070 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:10.903414 3070 heartbeater.cc:511] Master 127.6.46.254:37611 requested a full tablet report, sending...
I20260810 14:36:10.905480 2744 ts_manager.cc:194] Registered new tserver with Master: 08ca3ff693e64349b934f8f9727fe046 (127.6.46.193:44841)
I20260810 14:36:10.907440 2744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:55063
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:36:10.945220 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:37611
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:35293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:11.256500 3080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:11.256762 3080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:11.256867 3080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:11.263769 3080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:11.263953 3080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:11.264099 3080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:36:11.273336 3080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:35293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:37611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:11.274423 3080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:11.275627 3080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:11.287923 3088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:11.287884 3089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:11.288115 3091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:11.288946 3080 server_base.cc:1034] running on GCE node
I20260810 14:36:11.289445 3080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:11.290397 3080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:11.291587 3080 hybrid_clock.cc:648] HybridClock initialized: now 1786372571291538 us; error 44 us; skew 500 ppm
I20260810 14:36:11.291865 3080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:11 dist-test-slave-q9h9 krb5kdc[2675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:11.297778 3080 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:36:11.299197 3080 webserver.cc:460] Webserver started at http://127.6.46.194:36995/ using document root <none> and password file <none>
I20260810 14:36:11.299818 3080 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:11.299934 3080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:11.300237 3080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:11.303002 3080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "1b7e7f565e294410a3cde6af909c937a"
format_stamp: "Formatted at 2026-08-10 14:36:11 on dist-test-slave-q9h9"
server_key: "f9d411af5dc5e71f84d292871d85c2d5"
server_key_iv: "a18b5b0420501e452bbf4f388237e28a"
server_key_version: "encryptionkey@0"
I20260810 14:36:11.303791 3080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "1b7e7f565e294410a3cde6af909c937a"
format_stamp: "Formatted at 2026-08-10 14:36:11 on dist-test-slave-q9h9"
server_key: "f9d411af5dc5e71f84d292871d85c2d5"
server_key_iv: "a18b5b0420501e452bbf4f388237e28a"
server_key_version: "encryptionkey@0"
I20260810 14:36:11.308672 3080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 14:36:11.311985 3105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:11.313202 3080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:36:11.313325 3080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "1b7e7f565e294410a3cde6af909c937a"
format_stamp: "Formatted at 2026-08-10 14:36:11 on dist-test-slave-q9h9"
server_key: "f9d411af5dc5e71f84d292871d85c2d5"
server_key_iv: "a18b5b0420501e452bbf4f388237e28a"
server_key_version: "encryptionkey@0"
I20260810 14:36:11.313489 3080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:11.356879 3080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:11.360980 3080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:11.361264 3080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:11.362211 3080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:11.363473 3080 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:11.363546 3080 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:11.363656 3080 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:11.363706 3080 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:11.393430 3080 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:45085
I20260810 14:36:11.393481 3300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:45085 every 8 connection(s)
I20260810 14:36:11.394662 3080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:36:11.398077 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 3080
I20260810 14:36:11.398257 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:36:11.398856 6331 external_mini_cluster.cc:1428] Setting key d3fe3b8577efcd35aef8b8ad37afe8ff
Aug 10 14:36:11 dist-test-slave-q9h9 krb5kdc[2675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:36:11.415277 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.397001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:38011 (local address 127.6.46.254:37611)
0810 14:36:11.397168 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:11.397173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:11.398473 (+ 1300us) server_negotiation.cc:407] Connection header received
0810 14:36:11.400541 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:11.400545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:11.400620 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:11.400721 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:11.402624 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.403322 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.405306 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.405724 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.405813 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.409625 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:11.409631 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:11.409635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:11.409669 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:11.412167 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:11.412933 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:11.412937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:11.412939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:11.413067 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:11.413497 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:11.413500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:11.413502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:11.413768 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:11.413881 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:11.414688 (+ 807us) server_negotiation.cc:299] Negotiation successful
0810 14:36:11.414940 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:36:11.416198 3307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.397409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:37611 (local address 127.6.46.194:38011)
0810 14:36:11.398277 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:11.398341 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:11.400287 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:11.400958 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:11.400968 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:11.401683 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:11.402451 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.402476 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.403481 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.403487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.405180 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.405195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.405929 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.405936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.406223 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.406860 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:11.406892 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:11.409405 (+ 2513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:11.412297 (+ 2892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:11.412306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:11.412320 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:11.412831 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:11.413171 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:11.413175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:11.413178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:11.413401 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:11.413886 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:11.413894 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:11.414313 (+ 419us) client_negotiation.cc:769] Sending connection context
0810 14:36:11.414533 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 14:36:11.415029 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":300,"threads_started":1}
I20260810 14:36:11.417666 3302 heartbeater.cc:347] Connected to a master server at 127.6.46.254:37611
I20260810 14:36:11.417973 3302 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:11.418819 3302 heartbeater.cc:511] Master 127.6.46.254:37611 requested a full tablet report, sending...
I20260810 14:36:11.420408 2744 ts_manager.cc:194] Registered new tserver with Master: 1b7e7f565e294410a3cde6af909c937a (127.6.46.194:45085)
I20260810 14:36:11.421427 2744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:38011
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:36:11.462468 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:37611
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:35293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:11.786082 3311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:11.786341 3311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:11.786445 3311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:11.793367 3311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:11.793469 3311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:11.793576 3311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:36:11.803010 3311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:35293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:37611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:11.804106 3311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:11.805279 3311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:11.817531 3318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:11.817561 3324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:11.817584 3320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:11.818600 3311 server_base.cc:1034] running on GCE node
I20260810 14:36:11.819101 3311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:11.819890 3311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:11.821079 3311 hybrid_clock.cc:648] HybridClock initialized: now 1786372571821039 us; error 28 us; skew 500 ppm
I20260810 14:36:11.821357 3311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:11 dist-test-slave-q9h9 krb5kdc[2675](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:11.824244 3311 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:36:11.825578 3311 webserver.cc:460] Webserver started at http://127.6.46.195:38463/ using document root <none> and password file <none>
I20260810 14:36:11.826304 3311 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:11.826437 3311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:11.826773 3311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:11.829633 3311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "194de7b117a24fa38f078dd6dd2551c8"
format_stamp: "Formatted at 2026-08-10 14:36:11 on dist-test-slave-q9h9"
server_key: "955e15b2c012471522a64f3d50593e00"
server_key_iv: "208e852668576313adc9a3703e469fff"
server_key_version: "encryptionkey@0"
I20260810 14:36:11.830372 3311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "194de7b117a24fa38f078dd6dd2551c8"
format_stamp: "Formatted at 2026-08-10 14:36:11 on dist-test-slave-q9h9"
server_key: "955e15b2c012471522a64f3d50593e00"
server_key_iv: "208e852668576313adc9a3703e469fff"
server_key_version: "encryptionkey@0"
I20260810 14:36:11.835135 3311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 14:36:11.838506 3333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:11.839694 3311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260810 14:36:11.839830 3311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "194de7b117a24fa38f078dd6dd2551c8"
format_stamp: "Formatted at 2026-08-10 14:36:11 on dist-test-slave-q9h9"
server_key: "955e15b2c012471522a64f3d50593e00"
server_key_iv: "208e852668576313adc9a3703e469fff"
server_key_version: "encryptionkey@0"
I20260810 14:36:11.839993 3311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:11.857302 3311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:11.861241 3311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:11.861511 3311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:11.862402 3311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:11.863647 3311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:11.863724 3311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:11.863842 3311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:11.863896 3311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:11.894426 3311 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:44361
I20260810 14:36:11.894474 3516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:44361 every 8 connection(s)
I20260810 14:36:11.895576 3311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:36:11.898002 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 3311
I20260810 14:36:11.898221 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:36:11.898783 6331 external_mini_cluster.cc:1428] Setting key bf743f98ea386d3f088c65177a73142a
Aug 10 14:36:11 dist-test-slave-q9h9 krb5kdc[2675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372571, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:36:11.910056 3070 heartbeater.cc:503] Master 127.6.46.254:37611 was elected leader, sending a full tablet report...
I20260810 14:36:11.914521 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.897822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:56211 (local address 127.6.46.254:37611)
0810 14:36:11.897976 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:11.897981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:11.899423 (+ 1442us) server_negotiation.cc:407] Connection header received
0810 14:36:11.901418 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:11.901423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:11.901510 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:11.901602 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:11.903087 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.903819 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.905280 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.905691 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.905795 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.908904 (+ 3109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:11.908909 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:11.908913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:11.908944 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:11.911468 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:11.912289 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:11.912295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:11.912298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:11.912461 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:11.912881 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:11.912886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:11.912890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:11.913182 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:11.913320 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:11.913945 (+ 625us) server_negotiation.cc:299] Negotiation successful
0810 14:36:11.914191 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:36:11.915335 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.898492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:37611 (local address 127.6.46.195:56211)
0810 14:36:11.899267 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:11.899301 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:11.901149 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:11.901854 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:11.901861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:11.902401 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:11.902926 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.902942 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.903967 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.903970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.905141 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.905155 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.905901 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.905908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.906168 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.906727 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:11.906752 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:11.908712 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:11.911617 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:11.911625 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:11.911639 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:11.912159 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:11.912571 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:11.912577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:11.912580 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:11.912766 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:11.913310 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:11.913317 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:11.913597 (+ 280us) client_negotiation.cc:769] Sending connection context
0810 14:36:11.913830 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 14:36:11.914295 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":351,"threads_started":1}
I20260810 14:36:11.916458 3518 heartbeater.cc:347] Connected to a master server at 127.6.46.254:37611
I20260810 14:36:11.916729 3518 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:11.917544 3518 heartbeater.cc:511] Master 127.6.46.254:37611 requested a full tablet report, sending...
I20260810 14:36:11.918844 2744 ts_manager.cc:194] Registered new tserver with Master: 194de7b117a24fa38f078dd6dd2551c8 (127.6.46.195:44361)
I20260810 14:36:11.919574 2744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:56211
I20260810 14:36:11.925568 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:36:11.940924 3073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.930057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.6.46.254:37611)
0810 14:36:11.930199 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:11.930203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:11.930614 (+ 411us) server_negotiation.cc:407] Connection header received
0810 14:36:11.930742 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:11.930745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:11.930800 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:11.930874 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:11.932125 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.932810 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.933941 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.934375 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.934465 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.936615 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:11.936619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:11.936623 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:11.936650 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:11.938986 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:11.939497 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:11.939501 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:11.939503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:11.939618 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:11.939875 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:11.939877 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:11.939880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:11.940144 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:11.940332 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:11.940566 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 14:36:11.940713 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 14:36:11.944628 2744 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:36:11.951474 2744 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:36:11.973152 3543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.962518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:45085 (local address 127.0.0.1:44520)
0810 14:36:11.964237 (+ 1719us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:11.964292 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:11.964449 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:11.965445 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:11.965452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:11.965484 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:11.965848 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.965860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.967824 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.967830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.970153 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.970178 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.972235 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.972242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.972457 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.972505 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 14:36:11.972624 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 14:36:11.972792 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1506,"thread_start_us":1368,"threads_started":1}
I20260810 14:36:11.974144 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.962645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.6.46.194:45085)
0810 14:36:11.963739 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:11.963749 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:11.964618 (+ 869us) server_negotiation.cc:407] Connection header received
0810 14:36:11.964723 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:11.964731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:11.964963 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:11.965140 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:11.966052 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.967648 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.970348 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.971770 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.971897 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.972079 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:11.973617 (+ 1538us) server_negotiation.cc:299] Negotiation successful
0810 14:36:11.973796 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":915,"thread_start_us":1088,"threads_started":1}
I20260810 14:36:11.979164 3183 tablet_service.cc:1467] Processing CreateTablet for tablet edb54a9b5f024299b3e2f41a45c6b897 (DEFAULT_TABLE table=test-table [id=24b91e18b7ea4ed8bb250c137d7b56e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:11.980613 3183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edb54a9b5f024299b3e2f41a45c6b897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:11.980785 3543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.963299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:44361 (local address 127.0.0.1:40110)
0810 14:36:11.973366 (+ 10067us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:11.973382 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:11.973467 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:11.975687 (+ 2220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:11.975691 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:11.975716 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:11.975854 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.975865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.977269 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.977273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.978550 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.978564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.979971 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.979977 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.980219 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.980243 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 14:36:11.980321 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:36:11.980447 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9989,"mutex_wait_us":27,"thread_start_us":11113,"threads_started":1}
I20260810 14:36:11.980844 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.963474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.6.46.195:44361)
0810 14:36:11.975124 (+ 11650us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:11.975131 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:11.975166 (+ 35us) server_negotiation.cc:407] Connection header received
0810 14:36:11.975259 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:11.975266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:11.975443 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:11.975700 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:11.975965 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.977131 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.978736 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.979782 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.979942 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.980179 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:11.980379 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 14:36:11.980527 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11485,"thread_start_us":11330,"threads_started":1}
I20260810 14:36:11.982782 3546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.964595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:44841 (local address 127.0.0.1:37914)
0810 14:36:11.974677 (+ 10082us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:11.974699 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:11.974820 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:11.975882 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:11.975886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:11.975909 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:11.976215 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.976223 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.978265 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.978270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.980230 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:11.980245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.981953 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.981960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:11.982164 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.982197 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 14:36:11.982295 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 14:36:11.982412 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9972,"mutex_wait_us":3,"thread_start_us":10625,"threads_started":1}
I20260810 14:36:11.982824 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:11.965154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.6.46.193:44841)
0810 14:36:11.975331 (+ 10177us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:11.975338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:11.975365 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:36:11.975451 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:11.975457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:11.975687 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:11.976378 (+ 691us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:11.976626 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.978102 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.980407 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:11.981744 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:11.981881 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:11.982107 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:11.982346 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 14:36:11.982518 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10029,"thread_start_us":9884,"threads_started":1}
I20260810 14:36:11.984300 3402 tablet_service.cc:1467] Processing CreateTablet for tablet edb54a9b5f024299b3e2f41a45c6b897 (DEFAULT_TABLE table=test-table [id=24b91e18b7ea4ed8bb250c137d7b56e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:11.985387 3402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edb54a9b5f024299b3e2f41a45c6b897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:11.986372 2956 tablet_service.cc:1467] Processing CreateTablet for tablet edb54a9b5f024299b3e2f41a45c6b897 (DEFAULT_TABLE table=test-table [id=24b91e18b7ea4ed8bb250c137d7b56e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:11.987669 2956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet edb54a9b5f024299b3e2f41a45c6b897. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:11.995071 3551 tablet_bootstrap.cc:492] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a: Bootstrap starting.
I20260810 14:36:11.996480 3552 tablet_bootstrap.cc:492] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8: Bootstrap starting.
I20260810 14:36:11.998319 3551 tablet_bootstrap.cc:654] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:11.999313 3551 log.cc:824] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:11.999903 3552 tablet_bootstrap.cc:654] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:12.001175 3552 log.cc:824] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:12.001701 3556 tablet_bootstrap.cc:492] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046: Bootstrap starting.
I20260810 14:36:12.003103 3551 tablet_bootstrap.cc:492] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a: No bootstrap required, opened a new log
I20260810 14:36:12.003433 3551 ts_tablet_manager.cc:1397] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 14:36:12.004382 3552 tablet_bootstrap.cc:492] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8: No bootstrap required, opened a new log
I20260810 14:36:12.004611 3552 ts_tablet_manager.cc:1397] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260810 14:36:12.005086 3556 tablet_bootstrap.cc:654] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:12.006263 3556 log.cc:824] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:12.009066 3556 tablet_bootstrap.cc:492] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046: No bootstrap required, opened a new log
I20260810 14:36:12.009317 3556 ts_tablet_manager.cc:1397] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260810 14:36:12.012082 3552 raft_consensus.cc:348] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7e7f565e294410a3cde6af909c937a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45085 } } peers { permanent_uuid: "08ca3ff693e64349b934f8f9727fe046" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44841 } } peers { permanent_uuid: "194de7b117a24fa38f078dd6dd2551c8" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44361 } }
I20260810 14:36:12.012478 3552 raft_consensus.cc:374] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:12.012579 3552 raft_consensus.cc:729] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 194de7b117a24fa38f078dd6dd2551c8, State: Initialized, Role: FOLLOWER
I20260810 14:36:12.013154 3552 consensus_queue.cc:260] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7e7f565e294410a3cde6af909c937a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45085 } } peers { permanent_uuid: "08ca3ff693e64349b934f8f9727fe046" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44841 } } peers { permanent_uuid: "194de7b117a24fa38f078dd6dd2551c8" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44361 } }
I20260810 14:36:12.014221 3518 heartbeater.cc:503] Master 127.6.46.254:37611 was elected leader, sending a full tablet report...
I20260810 14:36:12.014734 3552 ts_tablet_manager.cc:1428] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8: Time spent starting tablet: real 0.010s user 0.004s sys 0.007s
I20260810 14:36:12.014878 3551 raft_consensus.cc:348] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7e7f565e294410a3cde6af909c937a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45085 } } peers { permanent_uuid: "08ca3ff693e64349b934f8f9727fe046" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44841 } } peers { permanent_uuid: "194de7b117a24fa38f078dd6dd2551c8" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44361 } }
I20260810 14:36:12.015298 3551 raft_consensus.cc:374] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:12.015398 3551 raft_consensus.cc:729] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7e7f565e294410a3cde6af909c937a, State: Initialized, Role: FOLLOWER
I20260810 14:36:12.015961 3551 consensus_queue.cc:260] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7e7f565e294410a3cde6af909c937a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45085 } } peers { permanent_uuid: "08ca3ff693e64349b934f8f9727fe046" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44841 } } peers { permanent_uuid: "194de7b117a24fa38f078dd6dd2551c8" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44361 } }
I20260810 14:36:12.017045 3302 heartbeater.cc:503] Master 127.6.46.254:37611 was elected leader, sending a full tablet report...
I20260810 14:36:12.017582 3551 ts_tablet_manager.cc:1428] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260810 14:36:12.019086 3556 raft_consensus.cc:348] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7e7f565e294410a3cde6af909c937a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45085 } } peers { permanent_uuid: "08ca3ff693e64349b934f8f9727fe046" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44841 } } peers { permanent_uuid: "194de7b117a24fa38f078dd6dd2551c8" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44361 } }
I20260810 14:36:12.019446 3556 raft_consensus.cc:374] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:12.019553 3556 raft_consensus.cc:729] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ca3ff693e64349b934f8f9727fe046, State: Initialized, Role: FOLLOWER
I20260810 14:36:12.020125 3556 consensus_queue.cc:260] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7e7f565e294410a3cde6af909c937a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45085 } } peers { permanent_uuid: "08ca3ff693e64349b934f8f9727fe046" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44841 } } peers { permanent_uuid: "194de7b117a24fa38f078dd6dd2551c8" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44361 } }
I20260810 14:36:12.021668 3556 ts_tablet_manager.cc:1428] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260810 14:36:12.115001 3561 raft_consensus.cc:482] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:36:12.115298 3561 raft_consensus.cc:504] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7e7f565e294410a3cde6af909c937a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45085 } } peers { permanent_uuid: "08ca3ff693e64349b934f8f9727fe046" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44841 } } peers { permanent_uuid: "194de7b117a24fa38f078dd6dd2551c8" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44361 } }
I20260810 14:36:12.117076 3561 leader_election.cc:288] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b7e7f565e294410a3cde6af909c937a (127.6.46.194:45085), 08ca3ff693e64349b934f8f9727fe046 (127.6.46.193:44841)
I20260810 14:36:12.122586 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:12.117396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:42097 (local address 127.6.46.194:45085)
0810 14:36:12.117592 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:12.117597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:12.117703 (+ 106us) server_negotiation.cc:407] Connection header received
0810 14:36:12.117922 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:12.117927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:12.118038 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:12.118136 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:12.118599 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.119315 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.120924 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.121720 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.121856 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:12.121931 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:12.122231 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 14:36:12.122358 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:36:12.122586 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:12.117407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:45085 (local address 127.6.46.195:42097)
0810 14:36:12.117608 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:12.117627 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:12.117768 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:12.118167 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:12.118171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:12.118201 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:12.118382 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:12.118391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.119460 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.119464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:12.120759 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:12.120771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.121872 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.121878 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:12.122075 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:12.122100 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 14:36:12.122179 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 14:36:12.122303 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 14:36:12.123669 3565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:12.117849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:44841 (local address 127.6.46.195:54645)
0810 14:36:12.118728 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:12.118742 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:12.118858 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:12.119147 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:12.119151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:12.119173 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:12.119554 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:12.119563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.120563 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.120567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:12.121848 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:12.121861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.123058 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.123064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:12.123218 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:12.123241 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:36:12.123315 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 14:36:12.123394 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"mutex_wait_us":25,"thread_start_us":544,"threads_started":1}
I20260810 14:36:12.123756 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:12.117963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:54645 (local address 127.6.46.193:44841)
0810 14:36:12.118218 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:12.118225 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:12.118825 (+ 600us) server_negotiation.cc:407] Connection header received
0810 14:36:12.118981 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:12.118986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:12.119043 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:12.119114 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:12.119730 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.120415 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.121994 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.122876 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.123017 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:12.123067 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:12.123376 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 14:36:12.123510 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 14:36:12.123643 3218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb54a9b5f024299b3e2f41a45c6b897" candidate_uuid: "194de7b117a24fa38f078dd6dd2551c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7e7f565e294410a3cde6af909c937a" is_pre_election: true
I20260810 14:36:12.124231 3218 raft_consensus.cc:2454] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 194de7b117a24fa38f078dd6dd2551c8 in term 0.
I20260810 14:36:12.124603 2994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb54a9b5f024299b3e2f41a45c6b897" candidate_uuid: "194de7b117a24fa38f078dd6dd2551c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ca3ff693e64349b934f8f9727fe046" is_pre_election: true
I20260810 14:36:12.124876 3336 leader_election.cc:302] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 194de7b117a24fa38f078dd6dd2551c8, 1b7e7f565e294410a3cde6af909c937a; no voters:
I20260810 14:36:12.125144 2994 raft_consensus.cc:2454] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 194de7b117a24fa38f078dd6dd2551c8 in term 0.
I20260810 14:36:12.125231 3561 raft_consensus.cc:2781] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:36:12.125345 3561 raft_consensus.cc:482] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:36:12.125442 3561 raft_consensus.cc:3037] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:12.127748 3561 raft_consensus.cc:504] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7e7f565e294410a3cde6af909c937a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45085 } } peers { permanent_uuid: "08ca3ff693e64349b934f8f9727fe046" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44841 } } peers { permanent_uuid: "194de7b117a24fa38f078dd6dd2551c8" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44361 } }
I20260810 14:36:12.128345 3561 leader_election.cc:288] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [CANDIDATE]: Term 1 election: Requested vote from peers 1b7e7f565e294410a3cde6af909c937a (127.6.46.194:45085), 08ca3ff693e64349b934f8f9727fe046 (127.6.46.193:44841)
I20260810 14:36:12.128768 3218 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb54a9b5f024299b3e2f41a45c6b897" candidate_uuid: "194de7b117a24fa38f078dd6dd2551c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7e7f565e294410a3cde6af909c937a"
I20260810 14:36:12.128836 2994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "edb54a9b5f024299b3e2f41a45c6b897" candidate_uuid: "194de7b117a24fa38f078dd6dd2551c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08ca3ff693e64349b934f8f9727fe046"
I20260810 14:36:12.128983 3218 raft_consensus.cc:3037] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:12.129062 2994 raft_consensus.cc:3037] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:12.131237 2994 raft_consensus.cc:2454] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 194de7b117a24fa38f078dd6dd2551c8 in term 1.
I20260810 14:36:12.131278 3218 raft_consensus.cc:2454] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 194de7b117a24fa38f078dd6dd2551c8 in term 1.
I20260810 14:36:12.131573 3336 leader_election.cc:302] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08ca3ff693e64349b934f8f9727fe046, 194de7b117a24fa38f078dd6dd2551c8; no voters:
I20260810 14:36:12.131834 3561 raft_consensus.cc:2781] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:12.132234 3561 raft_consensus.cc:686] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [term 1 LEADER]: Becoming Leader. State: Replica: 194de7b117a24fa38f078dd6dd2551c8, State: Running, Role: LEADER
I20260810 14:36:12.132733 3561 consensus_queue.cc:237] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7e7f565e294410a3cde6af909c937a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45085 } } peers { permanent_uuid: "08ca3ff693e64349b934f8f9727fe046" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44841 } } peers { permanent_uuid: "194de7b117a24fa38f078dd6dd2551c8" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44361 } }
W20260810 14:36:12.135604 3071 tablet.cc:2369] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:36:12.137542 2741 catalog_manager.cc:5495] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 194de7b117a24fa38f078dd6dd2551c8 (127.6.46.195). New cstate: current_term: 1 leader_uuid: "194de7b117a24fa38f078dd6dd2551c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7e7f565e294410a3cde6af909c937a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08ca3ff693e64349b934f8f9727fe046" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "194de7b117a24fa38f078dd6dd2551c8" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44361 } health_report { overall_health: HEALTHY } } }
W20260810 14:36:12.146489 3304 tablet.cc:2369] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:36:12.146929 3519 tablet.cc:2369] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:36:12.162132 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:12.156804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.6.46.195:44361)
0810 14:36:12.156970 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:12.156974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:12.157126 (+ 152us) server_negotiation.cc:407] Connection header received
0810 14:36:12.157290 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:12.157294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:12.157364 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:12.157482 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:36:12.157772 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.158556 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.160076 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.160530 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.160644 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:12.161146 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:36:12.161623 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:36:12.161737 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:12.161836 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 14:36:12.161905 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 14:36:12.169582 3218 raft_consensus.cc:1261] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a [term 1 FOLLOWER]: Refusing update from remote peer 194de7b117a24fa38f078dd6dd2551c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:12.169595 2994 raft_consensus.cc:1261] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046 [term 1 FOLLOWER]: Refusing update from remote peer 194de7b117a24fa38f078dd6dd2551c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:12.170452 3566 consensus_queue.cc:1035] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b7e7f565e294410a3cde6af909c937a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 45085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:12.170800 3561 consensus_queue.cc:1035] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08ca3ff693e64349b934f8f9727fe046" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:12.184252 3573 mvcc.cc:205] Tried to move back new op lower bound from 7316982055597334528 to 7316982055459639296. Current Snapshot: MvccSnapshot[applied={T|T < 7316982055597334528}]
I20260810 14:36:12.186739 3575 mvcc.cc:205] Tried to move back new op lower bound from 7316982055597334528 to 7316982055459639296. Current Snapshot: MvccSnapshot[applied={T|T < 7316982055597334528}]
I20260810 14:36:12.194188 2744 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:12.194417 2744 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42952:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:12.197665 2744 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 391bf123d6cc410e9e945f6c2a3fa028: Sending DeleteTablet for 3 replicas of tablet edb54a9b5f024299b3e2f41a45c6b897
I20260810 14:36:12.198822 3183 tablet_service.cc:1514] Processing DeleteTablet for tablet edb54a9b5f024299b3e2f41a45c6b897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:12 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:44520
I20260810 14:36:12.199141 2956 tablet_service.cc:1514] Processing DeleteTablet for tablet edb54a9b5f024299b3e2f41a45c6b897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:12 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:37914
I20260810 14:36:12.199504 3402 tablet_service.cc:1514] Processing DeleteTablet for tablet edb54a9b5f024299b3e2f41a45c6b897 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:12 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:40110
I20260810 14:36:12.199654 3588 tablet_replica.cc:330] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a: stopping tablet replica
I20260810 14:36:12.200157 3588 raft_consensus.cc:2229] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:12.200444 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 2849
I20260810 14:36:12.200691 3588 raft_consensus.cc:2258] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:12.202678 3588 ts_tablet_manager.cc:1905] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:12.209544 3588 ts_tablet_manager.cc:1918] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:12.209728 3588 log.cc:1196] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/edb54a9b5f024299b3e2f41a45c6b897
I20260810 14:36:12.210346 3588 ts_tablet_manager.cc:1939] T edb54a9b5f024299b3e2f41a45c6b897 P 1b7e7f565e294410a3cde6af909c937a: Deleting consensus metadata
I20260810 14:36:12.212010 2726 catalog_manager.cc:4841] TS 1b7e7f565e294410a3cde6af909c937a (127.6.46.194:45085): tablet edb54a9b5f024299b3e2f41a45c6b897 (table test-table [id=24b91e18b7ea4ed8bb250c137d7b56e5]) successfully deleted
I20260810 14:36:12.215142 3590 tablet_replica.cc:330] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8: stopping tablet replica
I20260810 14:36:12.215606 3590 raft_consensus.cc:2229] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:36:12.216445 3590 raft_consensus.cc:2258] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:12.217063 3589 tablet_replica.cc:330] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046: stopping tablet replica
I20260810 14:36:12.217407 3589 raft_consensus.cc:2229] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:12.217793 3589 raft_consensus.cc:2258] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:12.219731 3589 ts_tablet_manager.cc:1905] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:12.220209 3590 ts_tablet_manager.cc:1905] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 14:36:12 dist-test-slave-q9h9 krb5kdc[2675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:36:12.226219 3589 ts_tablet_manager.cc:1918] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:12.226375 3589 log.cc:1196] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/edb54a9b5f024299b3e2f41a45c6b897
I20260810 14:36:12.226986 3589 ts_tablet_manager.cc:1939] T edb54a9b5f024299b3e2f41a45c6b897 P 08ca3ff693e64349b934f8f9727fe046: Deleting consensus metadata
I20260810 14:36:12.228114 3590 ts_tablet_manager.cc:1918] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:12.228319 3590 log.cc:1196] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/edb54a9b5f024299b3e2f41a45c6b897
I20260810 14:36:12.228756 2726 catalog_manager.cc:4841] TS 08ca3ff693e64349b934f8f9727fe046 (127.6.46.193:44841): tablet edb54a9b5f024299b3e2f41a45c6b897 (table test-table [id=24b91e18b7ea4ed8bb250c137d7b56e5]) successfully deleted
I20260810 14:36:12.228978 3590 ts_tablet_manager.cc:1939] T edb54a9b5f024299b3e2f41a45c6b897 P 194de7b117a24fa38f078dd6dd2551c8: Deleting consensus metadata
I20260810 14:36:12.230415 2725 catalog_manager.cc:4841] TS 194de7b117a24fa38f078dd6dd2551c8 (127.6.46.195:44361): tablet edb54a9b5f024299b3e2f41a45c6b897 (table test-table [id=24b91e18b7ea4ed8bb250c137d7b56e5]) successfully deleted
I20260810 14:36:12.233134 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:12.201844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.6.46.193:44841)
0810 14:36:12.202051 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:12.202058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:12.214490 (+ 12432us) server_negotiation.cc:407] Connection header received
0810 14:36:12.214675 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:12.214680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:12.214786 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:12.214890 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:12.216780 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.218049 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.220042 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.220733 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.220840 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:12.226269 (+ 5429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:12.226284 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:12.226301 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:12.226346 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:12.230370 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:12.231107 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:12.231116 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:12.231120 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:12.231296 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:12.231621 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:12.231625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:12.231628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:12.232064 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:12.232373 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:12.232555 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 14:36:12.232768 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:36:12.233767 3061 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:12.333783 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 2849
I20260810 14:36:12.357034 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 3080
Aug 10 14:36:12 dist-test-slave-q9h9 krb5kdc[2675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:36:12.373682 3544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:12.357866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.6.46.194:45085)
0810 14:36:12.358108 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:12.358115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:12.358137 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:36:12.358225 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:12.358231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:12.358316 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:12.358435 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:12.359639 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.360796 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.362238 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.362873 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.363006 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:12.367318 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:12.367338 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:12.367359 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:12.367409 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:12.370945 (+ 3536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:12.371615 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:12.371625 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:12.371629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:12.371782 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:12.372102 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:12.372107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:12.372111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:12.372605 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:12.372912 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:12.373077 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 14:36:12.373310 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 14:36:12.374477 3294 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:12.461117 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 3080
I20260810 14:36:12.481590 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 3311
Aug 10 14:36:12 dist-test-slave-q9h9 krb5kdc[2675](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372569, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:36:12.499261 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:12.482344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.6.46.195:44361)
0810 14:36:12.482677 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:12.482684 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:12.482733 (+ 49us) server_negotiation.cc:407] Connection header received
0810 14:36:12.482809 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:12.482814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:12.482904 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:12.483020 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:12.484278 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.485492 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.486901 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:12.487581 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:12.487725 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:12.492319 (+ 4594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:12.492333 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:12.492353 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:12.492401 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:12.496375 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:12.497020 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:12.497029 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:12.497032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:12.497237 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:12.497580 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:12.497585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:12.497588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:12.498082 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:12.498445 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:12.498617 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 14:36:12.498842 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260810 14:36:12.500110 3510 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:12.618316 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 3311
I20260810 14:36:12.645128 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 2698
I20260810 14:36:12.646297 2802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:12.753736 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 2698
2026-08-10T14:36:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4928 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T14:36:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:12Z Disabled control of system clock
I20260810 14:36:12.840407 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36109
--webserver_interface=127.6.46.254
--webserver_port=0
--builtin_ntp_servers=127.6.46.212:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:36109
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:36:13.156206 3605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:13.156468 3605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:13.156545 3605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:13.163482 3605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:13.163576 3605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:13.163619 3605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:13.163674 3605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:13.163717 3605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:13.172487 3605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:36109
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:36109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:13.173601 3605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:13.175346 3605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:13.186666 3622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:13.186702 3617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:13.186736 3615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:13.187674 3605 server_base.cc:1034] running on GCE node
I20260810 14:36:13.188337 3605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:13.189647 3605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:13.190840 3605 hybrid_clock.cc:648] HybridClock initialized: now 1786372573190789 us; error 43 us; skew 500 ppm
I20260810 14:36:13.191175 3605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:36:13.193246 3605 webserver.cc:460] Webserver started at http://127.6.46.254:41097/ using document root <none> and password file <none>
I20260810 14:36:13.193892 3605 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:13.193992 3605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:13.194353 3605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:13.197111 3605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "c7667e3074634484aeac393b876dc231"
format_stamp: "Formatted at 2026-08-10 14:36:13 on dist-test-slave-q9h9"
server_key: "09591b412a6c91ee2466e2307c8f8f97"
server_key_iv: "8b79909691fdeb2cf1adf75671f262ad"
server_key_version: "encryptionkey@0"
I20260810 14:36:13.197861 3605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "c7667e3074634484aeac393b876dc231"
format_stamp: "Formatted at 2026-08-10 14:36:13 on dist-test-slave-q9h9"
server_key: "09591b412a6c91ee2466e2307c8f8f97"
server_key_iv: "8b79909691fdeb2cf1adf75671f262ad"
server_key_version: "encryptionkey@0"
I20260810 14:36:13.202699 3605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 14:36:13.205899 3631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:13.207165 3605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:36:13.207340 3605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "c7667e3074634484aeac393b876dc231"
format_stamp: "Formatted at 2026-08-10 14:36:13 on dist-test-slave-q9h9"
server_key: "09591b412a6c91ee2466e2307c8f8f97"
server_key_iv: "8b79909691fdeb2cf1adf75671f262ad"
server_key_version: "encryptionkey@0"
I20260810 14:36:13.207530 3605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:13.220688 3605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:13.221391 3605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:13.221642 3605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:13.238569 3605 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:36109
I20260810 14:36:13.238591 3718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:36109 every 8 connection(s)
I20260810 14:36:13.239799 3605 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:13.240558 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 3605
I20260810 14:36:13.240762 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:13.241227 6331 external_mini_cluster.cc:1428] Setting key 2373316b0046bbc40e4cc81a56a5a5bd
I20260810 14:36:13.243927 3719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:13.256373 3725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:13.243000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.6.46.254:36109)
0810 14:36:13.244054 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:13.244068 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:13.244119 (+ 51us) server_negotiation.cc:407] Connection header received
0810 14:36:13.246364 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:13.246391 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:13.246949 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:13.247377 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:13.247893 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.249968 (+ 2075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.251359 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.252354 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.252534 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:13.253567 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:13.253577 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:13.253598 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:13.254361 (+ 763us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:13.254455 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:13.254482 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:13.254608 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:13.254843 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 14:36:13.255187 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":347,"threads_started":1}
I20260810 14:36:13.257388 3719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231: Bootstrap starting.
I20260810 14:36:13.260744 3719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:13.261821 3719 log.cc:824] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:13.264307 3719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231: No bootstrap required, opened a new log
I20260810 14:36:13.271354 3719 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7667e3074634484aeac393b876dc231" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36109 } }
I20260810 14:36:13.271668 3719 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:13.271765 3719 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7667e3074634484aeac393b876dc231, State: Initialized, Role: FOLLOWER
I20260810 14:36:13.272264 3719 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7667e3074634484aeac393b876dc231" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36109 } }
I20260810 14:36:13.272473 3719 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:13.272604 3719 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:13.272770 3719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:13.274976 3719 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7667e3074634484aeac393b876dc231" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36109 } }
I20260810 14:36:13.275434 3719 leader_election.cc:302] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7667e3074634484aeac393b876dc231; no voters:
I20260810 14:36:13.275972 3719 leader_election.cc:288] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:13.276108 3727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:13.276785 3727 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [term 1 LEADER]: Becoming Leader. State: Replica: c7667e3074634484aeac393b876dc231, State: Running, Role: LEADER
I20260810 14:36:13.277287 3727 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7667e3074634484aeac393b876dc231" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36109 } }
I20260810 14:36:13.277777 3719 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:13.279937 3729 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7667e3074634484aeac393b876dc231. Latest consensus state: current_term: 1 leader_uuid: "c7667e3074634484aeac393b876dc231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7667e3074634484aeac393b876dc231" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36109 } } }
I20260810 14:36:13.279886 3728 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7667e3074634484aeac393b876dc231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7667e3074634484aeac393b876dc231" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 36109 } } }
I20260810 14:36:13.280325 3728 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:13.280329 3729 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:13.283039 3744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:13.287720 3744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:13.295809 3744 catalog_manager.cc:1347] Generated new cluster ID: 5d880d1019a548c9bad140627571a231
I20260810 14:36:13.295912 3744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:13.304806 3744 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:13.305950 3744 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:13.321692 3744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231: Generated new TSK 0
I20260810 14:36:13.322481 3744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:13.335184 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36109
--builtin_ntp_servers=127.6.46.212:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:36:13.664004 3762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:13.664265 3762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:13.664361 3762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:13.671232 3762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:13.671315 3762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:13.671424 3762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:36:13.680147 3762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:13.681331 3762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:13.683038 3762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:13.695057 3774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:13.695091 3772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:13.695012 3771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:13.696063 3762 server_base.cc:1034] running on GCE node
I20260810 14:36:13.696709 3762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:13.697706 3762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:13.698894 3762 hybrid_clock.cc:648] HybridClock initialized: now 1786372573698848 us; error 40 us; skew 500 ppm
I20260810 14:36:13.699178 3762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:36:13.701304 3762 webserver.cc:460] Webserver started at http://127.6.46.193:37917/ using document root <none> and password file <none>
I20260810 14:36:13.701922 3762 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:13.702042 3762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:13.702343 3762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:13.704967 3762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "f612521b56714c43b8041767379cef94"
format_stamp: "Formatted at 2026-08-10 14:36:13 on dist-test-slave-q9h9"
server_key: "0682c4a02c17d7efbb23d1c84c2c95cb"
server_key_iv: "801900b41e691af8adf118cf18da97c6"
server_key_version: "encryptionkey@0"
I20260810 14:36:13.705706 3762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "f612521b56714c43b8041767379cef94"
format_stamp: "Formatted at 2026-08-10 14:36:13 on dist-test-slave-q9h9"
server_key: "0682c4a02c17d7efbb23d1c84c2c95cb"
server_key_iv: "801900b41e691af8adf118cf18da97c6"
server_key_version: "encryptionkey@0"
I20260810 14:36:13.711238 3762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260810 14:36:13.714452 3783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:13.715651 3762 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260810 14:36:13.715801 3762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "f612521b56714c43b8041767379cef94"
format_stamp: "Formatted at 2026-08-10 14:36:13 on dist-test-slave-q9h9"
server_key: "0682c4a02c17d7efbb23d1c84c2c95cb"
server_key_iv: "801900b41e691af8adf118cf18da97c6"
server_key_version: "encryptionkey@0"
I20260810 14:36:13.715991 3762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:13.728343 3762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:13.729015 3762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:13.729281 3762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:13.730152 3762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:13.731460 3762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:13.731567 3762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:13.731704 3762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:13.731771 3762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:13.760794 3762 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:37091
I20260810 14:36:13.760848 3973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:37091 every 8 connection(s)
I20260810 14:36:13.761940 3762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:36:13.772114 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 3762
I20260810 14:36:13.772320 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:36:13.772795 6331 external_mini_cluster.cc:1428] Setting key 2ca8ee8a063dfdc59109fbe26606bfe1
I20260810 14:36:13.773531 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:13.764305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:56145 (local address 127.6.46.254:36109)
0810 14:36:13.764723 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:13.764728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:13.765807 (+ 1079us) server_negotiation.cc:407] Connection header received
0810 14:36:13.767511 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:13.767517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:13.767602 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:13.767699 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:13.768692 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.769407 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.770939 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.771353 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.771465 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:13.772781 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:13.772789 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:13.772793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:13.772814 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:13.772874 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:13.772892 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:13.772991 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:13.773173 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 14:36:13.773269 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":213,"threads_started":1}
I20260810 14:36:13.774616 3978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:13.764830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36109 (local address 127.6.46.193:56145)
0810 14:36:13.765652 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:13.765685 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:13.767246 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:13.767927 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:13.767935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:13.768016 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:13.768526 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:13.768541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.769555 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.769559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:13.770793 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:13.770806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:13.771568 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:13.771576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:13.771822 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:13.772486 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:13.772489 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:13.772523 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:13.772591 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:13.772987 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:13.772996 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:13.773002 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:36:13.773134 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 14:36:13.773430 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":268,"threads_started":1}
I20260810 14:36:13.775800 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36109
--builtin_ntp_servers=127.6.46.212:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:36:13.776118 3974 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36109
I20260810 14:36:13.776373 3974 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:13.777019 3974 heartbeater.cc:511] Master 127.6.46.254:36109 requested a full tablet report, sending...
I20260810 14:36:13.779084 3657 ts_manager.cc:194] Registered new tserver with Master: f612521b56714c43b8041767379cef94 (127.6.46.193:37091)
I20260810 14:36:13.780881 3657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.46.193:56145
W20260810 14:36:14.093777 3979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:14.094054 3979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:14.094190 3979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:14.101102 3979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:14.101197 3979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:14.101332 3979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:36:14.109980 3979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:14.111230 3979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:14.113027 3979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:14.125888 3989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:14.125844 3994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:14.125932 3990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:14.126958 3979 server_base.cc:1034] running on GCE node
I20260810 14:36:14.127490 3979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:14.128602 3979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:14.129837 3979 hybrid_clock.cc:648] HybridClock initialized: now 1786372574129786 us; error 32 us; skew 500 ppm
I20260810 14:36:14.130200 3979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:36:14.132333 3979 webserver.cc:460] Webserver started at http://127.6.46.194:41265/ using document root <none> and password file <none>
I20260810 14:36:14.132962 3979 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:14.133056 3979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:14.133492 3979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:14.136274 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "28a7b542ea19461db1d56e00cc76191f"
format_stamp: "Formatted at 2026-08-10 14:36:14 on dist-test-slave-q9h9"
server_key: "f8143eb838ab8ecc09866d5bee66c93b"
server_key_iv: "a2f72346852ee787c3d79b4e5f203c13"
server_key_version: "encryptionkey@0"
I20260810 14:36:14.137002 3979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "28a7b542ea19461db1d56e00cc76191f"
format_stamp: "Formatted at 2026-08-10 14:36:14 on dist-test-slave-q9h9"
server_key: "f8143eb838ab8ecc09866d5bee66c93b"
server_key_iv: "a2f72346852ee787c3d79b4e5f203c13"
server_key_version: "encryptionkey@0"
I20260810 14:36:14.142040 3979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 14:36:14.145387 4005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:14.146710 3979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:14.146924 3979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "28a7b542ea19461db1d56e00cc76191f"
format_stamp: "Formatted at 2026-08-10 14:36:14 on dist-test-slave-q9h9"
server_key: "f8143eb838ab8ecc09866d5bee66c93b"
server_key_iv: "a2f72346852ee787c3d79b4e5f203c13"
server_key_version: "encryptionkey@0"
I20260810 14:36:14.147149 3979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:14.172122 3979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:14.172838 3979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:14.173115 3979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:14.174001 3979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:14.175367 3979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:14.175460 3979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:14.175561 3979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:14.175635 3979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:14.204968 3979 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:42227
I20260810 14:36:14.205030 4180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:42227 every 8 connection(s)
I20260810 14:36:14.206193 3979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:36:14.213397 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 3979
I20260810 14:36:14.213577 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:36:14.214170 6331 external_mini_cluster.cc:1428] Setting key d23e14921281a4e623ac4771c44ce311
I20260810 14:36:14.216796 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36109
--builtin_ntp_servers=127.6.46.212:32989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:36:14.218214 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.208480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:43365 (local address 127.6.46.254:36109)
0810 14:36:14.208694 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:14.208701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:14.209961 (+ 1260us) server_negotiation.cc:407] Connection header received
0810 14:36:14.211734 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:14.211740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:14.211820 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:14.211910 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:14.212864 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.213855 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.215511 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.216049 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.216173 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.217406 (+ 1233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:14.217411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:14.217414 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:14.217431 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:14.217470 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:14.217482 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:14.217559 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:14.217770 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 14:36:14.217866 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 14:36:14.219203 4184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.208969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36109 (local address 127.6.46.194:43365)
0810 14:36:14.209816 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:14.209852 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:14.211470 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:14.212105 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:14.212112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:14.212199 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:14.212736 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.212752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.213999 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.214003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.215373 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.215385 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.216263 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.216271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.216485 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.217122 (+ 637us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:14.217125 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:14.217164 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:14.217236 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:14.217569 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:14.217580 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:14.217588 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 14:36:14.217742 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 14:36:14.218119 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":265,"threads_started":1}
I20260810 14:36:14.220508 4181 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36109
I20260810 14:36:14.220774 4181 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:14.221431 4181 heartbeater.cc:511] Master 127.6.46.254:36109 requested a full tablet report, sending...
I20260810 14:36:14.222783 3657 ts_manager.cc:194] Registered new tserver with Master: 28a7b542ea19461db1d56e00cc76191f (127.6.46.194:42227)
I20260810 14:36:14.223735 3657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.46.194:43365
W20260810 14:36:14.535370 4186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:14.535641 4186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:14.535769 4186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:14.542622 4186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:14.542722 4186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:14.542862 4186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:36:14.551465 4186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:32989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:36109
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:14.552572 4186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:14.554265 4186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:14.566179 4192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:14.566236 4193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:14.566349 4196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:14.567073 4186 server_base.cc:1034] running on GCE node
I20260810 14:36:14.567610 4186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:14.568452 4186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:14.569648 4186 hybrid_clock.cc:648] HybridClock initialized: now 1786372574569599 us; error 38 us; skew 500 ppm
I20260810 14:36:14.569943 4186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:36:14.572067 4186 webserver.cc:460] Webserver started at http://127.6.46.195:34921/ using document root <none> and password file <none>
I20260810 14:36:14.572691 4186 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:14.572793 4186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:14.573101 4186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:14.575848 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "983642e8c1ab4c4c8985b9c5f433b616"
format_stamp: "Formatted at 2026-08-10 14:36:14 on dist-test-slave-q9h9"
server_key: "996df28443ad13757592b93fa4173401"
server_key_iv: "c7d475d539bd16d862593f7e0b74e7b7"
server_key_version: "encryptionkey@0"
I20260810 14:36:14.576584 4186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "983642e8c1ab4c4c8985b9c5f433b616"
format_stamp: "Formatted at 2026-08-10 14:36:14 on dist-test-slave-q9h9"
server_key: "996df28443ad13757592b93fa4173401"
server_key_iv: "c7d475d539bd16d862593f7e0b74e7b7"
server_key_version: "encryptionkey@0"
I20260810 14:36:14.581310 4186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:36:14.584615 4208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:14.585749 4186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:36:14.585896 4186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "983642e8c1ab4c4c8985b9c5f433b616"
format_stamp: "Formatted at 2026-08-10 14:36:14 on dist-test-slave-q9h9"
server_key: "996df28443ad13757592b93fa4173401"
server_key_iv: "c7d475d539bd16d862593f7e0b74e7b7"
server_key_version: "encryptionkey@0"
I20260810 14:36:14.586120 4186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:14.597200 4186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:14.597872 4186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:14.598181 4186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:14.599030 4186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:14.600309 4186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:14.600402 4186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:14.600505 4186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:14.600579 4186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:14.628834 4186 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:32965
I20260810 14:36:14.628891 4388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:32965 every 8 connection(s)
I20260810 14:36:14.630003 4186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:36:14.634415 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 4186
I20260810 14:36:14.634572 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:36:14.635011 6331 external_mini_cluster.cc:1428] Setting key b347d8ae6987395f5fb893158e3d1e2b
I20260810 14:36:14.641450 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.632214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:33641 (local address 127.6.46.254:36109)
0810 14:36:14.632380 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:14.632385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:14.633642 (+ 1257us) server_negotiation.cc:407] Connection header received
0810 14:36:14.635504 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:14.635510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:14.635596 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:14.635704 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:14.636722 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.637448 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.638973 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.639383 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.639493 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.640641 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:14.640646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:14.640649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:14.640674 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:14.640707 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:14.640718 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:14.640792 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:14.641031 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 14:36:14.641131 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:36:14.642438 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.632609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:36109 (local address 127.6.46.195:33641)
0810 14:36:14.633497 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:14.633531 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:14.635245 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:14.635912 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:14.635920 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:14.636010 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:14.636538 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.636554 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.637601 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.637605 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.638834 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.638847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.639592 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.639600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.639797 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.640358 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:14.640360 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:14.640400 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:14.640468 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:14.640809 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:14.640821 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:14.640827 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:36:14.640979 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 14:36:14.641343 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":279,"threads_started":1}
I20260810 14:36:14.643664 4390 heartbeater.cc:347] Connected to a master server at 127.6.46.254:36109
I20260810 14:36:14.643915 4390 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:14.644552 4390 heartbeater.cc:511] Master 127.6.46.254:36109 requested a full tablet report, sending...
I20260810 14:36:14.645823 3657 ts_manager.cc:194] Registered new tserver with Master: 983642e8c1ab4c4c8985b9c5f433b616 (127.6.46.195:32965)
I20260810 14:36:14.646613 3657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.46.195:33641
I20260810 14:36:14.649503 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:36:14.659364 3977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.653818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.6.46.254:36109)
0810 14:36:14.654049 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:14.654056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:14.654502 (+ 446us) server_negotiation.cc:407] Connection header received
0810 14:36:14.654605 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:14.654609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:14.654703 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:14.654793 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:14.655237 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.655918 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.657310 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.657736 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.657833 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.658826 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:14.658831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:14.658834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:14.658851 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:14.658883 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:14.658896 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:14.658964 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:14.659047 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 14:36:14.659122 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 14:36:14.662761 3657 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:36:14.669312 3657 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:36:14.689956 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.680169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:42227 (local address 127.0.0.1:58206)
0810 14:36:14.681114 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:14.681179 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:14.681355 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:14.682618 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:14.682646 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:14.682679 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:14.683056 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.683068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.685217 (+ 2149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.685222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.687440 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.687459 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.688821 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.688829 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.689376 (+ 547us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.689442 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 14:36:14.689558 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 14:36:14.689708 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"mutex_wait_us":327,"thread_start_us":1302,"threads_started":1}
I20260810 14:36:14.691056 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.680541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58206 (local address 127.6.46.194:42227)
0810 14:36:14.681954 (+ 1413us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:14.681962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:14.681992 (+ 30us) server_negotiation.cc:407] Connection header received
0810 14:36:14.682112 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:14.682120 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:14.682387 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:14.683307 (+ 920us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:14.683599 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.684940 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.687623 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.688582 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.689029 (+ 447us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.690428 (+ 1399us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:14.690500 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 14:36:14.690685 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1253,"thread_start_us":1124,"threads_started":1}
I20260810 14:36:14.691272 4420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.680373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:32965 (local address 127.0.0.1:34860)
0810 14:36:14.682290 (+ 1917us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:14.682312 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:14.682423 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:14.684299 (+ 1876us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:14.684303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:14.684328 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:14.684738 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.684750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.686450 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.686456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.688516 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.688531 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.690528 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.690534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.690690 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.690721 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 14:36:14.690814 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 14:36:14.690929 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1783,"mutex_wait_us":1,"thread_start_us":3178,"threads_started":1}
I20260810 14:36:14.691756 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.681083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34860 (local address 127.6.46.195:32965)
0810 14:36:14.683689 (+ 2606us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:14.683697 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:14.683731 (+ 34us) server_negotiation.cc:407] Connection header received
0810 14:36:14.683829 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:14.683836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:14.684099 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:14.684293 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:14.684944 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.686322 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.688747 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.690118 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.690254 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.690504 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:14.690884 (+ 380us) server_negotiation.cc:299] Negotiation successful
0810 14:36:14.691044 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2433,"thread_start_us":2286,"threads_started":1}
I20260810 14:36:14.692905 4418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.680777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:37091 (local address 127.0.0.1:40500)
0810 14:36:14.683667 (+ 2890us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:14.683687 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:14.683799 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:14.684545 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:14.684549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:14.684571 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:14.685125 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.685139 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.687037 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.687043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.689142 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.689157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.691237 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.691243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.691421 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.691450 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 14:36:14.692573 (+ 1123us) client_negotiation.cc:240] Negotiation successful
0810 14:36:14.692697 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2767,"mutex_wait_us":1,"thread_start_us":1260,"threads_started":1}
I20260810 14:36:14.693508 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.681294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.6.46.193:37091)
0810 14:36:14.683303 (+ 2009us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:14.683312 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:14.683894 (+ 582us) server_negotiation.cc:407] Connection header received
0810 14:36:14.683980 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:14.683985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:14.684135 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:14.684306 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:14.685349 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.686889 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.689371 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.690749 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.690844 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.691047 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:14.693150 (+ 2103us) server_negotiation.cc:299] Negotiation successful
0810 14:36:14.693270 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1848,"thread_start_us":1717,"threads_started":1}
I20260810 14:36:14.695362 4293 tablet_service.cc:1467] Processing CreateTablet for tablet c42efd5185894d9b8d93a9e62dbd35c2 (DEFAULT_TABLE table=test-table [id=80d2ce1ed49c490c8074de76959836af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:14.695362 4075 tablet_service.cc:1467] Processing CreateTablet for tablet c42efd5185894d9b8d93a9e62dbd35c2 (DEFAULT_TABLE table=test-table [id=80d2ce1ed49c490c8074de76959836af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:14.696508 3865 tablet_service.cc:1467] Processing CreateTablet for tablet c42efd5185894d9b8d93a9e62dbd35c2 (DEFAULT_TABLE table=test-table [id=80d2ce1ed49c490c8074de76959836af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:14.696864 4075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42efd5185894d9b8d93a9e62dbd35c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:14.696864 4293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42efd5185894d9b8d93a9e62dbd35c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:14.697516 3865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42efd5185894d9b8d93a9e62dbd35c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:14.708324 4427 tablet_bootstrap.cc:492] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94: Bootstrap starting.
I20260810 14:36:14.711988 4427 tablet_bootstrap.cc:654] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:14.713702 4427 log.cc:824] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:14.714726 4428 tablet_bootstrap.cc:492] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616: Bootstrap starting.
I20260810 14:36:14.716225 4429 tablet_bootstrap.cc:492] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f: Bootstrap starting.
I20260810 14:36:14.718436 4428 tablet_bootstrap.cc:654] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:14.719233 4429 tablet_bootstrap.cc:654] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:14.719640 4428 log.cc:824] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:14.720341 4429 log.cc:824] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:14.720898 4427 tablet_bootstrap.cc:492] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94: No bootstrap required, opened a new log
I20260810 14:36:14.721493 4427 ts_tablet_manager.cc:1397] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.011s
I20260810 14:36:14.723246 4429 tablet_bootstrap.cc:492] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f: No bootstrap required, opened a new log
I20260810 14:36:14.723285 4428 tablet_bootstrap.cc:492] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616: No bootstrap required, opened a new log
I20260810 14:36:14.723510 4428 ts_tablet_manager.cc:1397] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260810 14:36:14.723512 4429 ts_tablet_manager.cc:1397] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 14:36:14.730257 4428 raft_consensus.cc:348] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7b542ea19461db1d56e00cc76191f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42227 } } peers { permanent_uuid: "983642e8c1ab4c4c8985b9c5f433b616" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 32965 } } peers { permanent_uuid: "f612521b56714c43b8041767379cef94" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 37091 } }
I20260810 14:36:14.730592 4428 raft_consensus.cc:374] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:14.730701 4428 raft_consensus.cc:729] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983642e8c1ab4c4c8985b9c5f433b616, State: Initialized, Role: FOLLOWER
I20260810 14:36:14.731251 4428 consensus_queue.cc:260] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7b542ea19461db1d56e00cc76191f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42227 } } peers { permanent_uuid: "983642e8c1ab4c4c8985b9c5f433b616" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 32965 } } peers { permanent_uuid: "f612521b56714c43b8041767379cef94" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 37091 } }
I20260810 14:36:14.732237 4390 heartbeater.cc:503] Master 127.6.46.254:36109 was elected leader, sending a full tablet report...
I20260810 14:36:14.732856 4428 ts_tablet_manager.cc:1428] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20260810 14:36:14.731964 4427 raft_consensus.cc:348] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7b542ea19461db1d56e00cc76191f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42227 } } peers { permanent_uuid: "983642e8c1ab4c4c8985b9c5f433b616" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 32965 } } peers { permanent_uuid: "f612521b56714c43b8041767379cef94" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 37091 } }
I20260810 14:36:14.733501 4427 raft_consensus.cc:374] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:14.733302 4429 raft_consensus.cc:348] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7b542ea19461db1d56e00cc76191f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42227 } } peers { permanent_uuid: "983642e8c1ab4c4c8985b9c5f433b616" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 32965 } } peers { permanent_uuid: "f612521b56714c43b8041767379cef94" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 37091 } }
I20260810 14:36:14.733603 4427 raft_consensus.cc:729] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f612521b56714c43b8041767379cef94, State: Initialized, Role: FOLLOWER
I20260810 14:36:14.733681 4429 raft_consensus.cc:374] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:14.733768 4429 raft_consensus.cc:729] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a7b542ea19461db1d56e00cc76191f, State: Initialized, Role: FOLLOWER
I20260810 14:36:14.734221 4427 consensus_queue.cc:260] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7b542ea19461db1d56e00cc76191f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42227 } } peers { permanent_uuid: "983642e8c1ab4c4c8985b9c5f433b616" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 32965 } } peers { permanent_uuid: "f612521b56714c43b8041767379cef94" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 37091 } }
I20260810 14:36:14.734606 4429 consensus_queue.cc:260] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7b542ea19461db1d56e00cc76191f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42227 } } peers { permanent_uuid: "983642e8c1ab4c4c8985b9c5f433b616" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 32965 } } peers { permanent_uuid: "f612521b56714c43b8041767379cef94" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 37091 } }
I20260810 14:36:14.735924 4427 ts_tablet_manager.cc:1428] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260810 14:36:14.736065 3974 heartbeater.cc:503] Master 127.6.46.254:36109 was elected leader, sending a full tablet report...
I20260810 14:36:14.736809 4181 heartbeater.cc:503] Master 127.6.46.254:36109 was elected leader, sending a full tablet report...
I20260810 14:36:14.737411 4429 ts_tablet_manager.cc:1428] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260810 14:36:14.763934 3975 tablet.cc:2369] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:36:14.769505 4434 raft_consensus.cc:482] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:36:14.769762 4434 raft_consensus.cc:504] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7b542ea19461db1d56e00cc76191f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42227 } } peers { permanent_uuid: "983642e8c1ab4c4c8985b9c5f433b616" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 32965 } } peers { permanent_uuid: "f612521b56714c43b8041767379cef94" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 37091 } }
I20260810 14:36:14.771349 4434 leader_election.cc:288] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28a7b542ea19461db1d56e00cc76191f (127.6.46.194:42227), f612521b56714c43b8041767379cef94 (127.6.46.193:37091)
I20260810 14:36:14.776638 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.771903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:54759 (local address 127.6.46.193:37091)
0810 14:36:14.772121 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:14.772127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:14.772145 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:36:14.772212 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:14.772215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:14.772267 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:14.772347 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:14.772743 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.773423 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.775094 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.775835 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.775972 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.776040 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:14.776291 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 14:36:14.776411 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260810 14:36:14.776679 4393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.771587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:37091 (local address 127.6.46.195:54759)
0810 14:36:14.771756 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:14.771776 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:14.771904 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:14.772406 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:14.772411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:14.772442 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:14.772601 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.772609 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.773550 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.773556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.774941 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.774954 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.775982 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.775987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.776139 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.776171 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 14:36:14.776247 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 14:36:14.776384 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 14:36:14.777220 4438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.771735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:42227 (local address 127.6.46.195:52777)
0810 14:36:14.772435 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:14.772449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:14.772577 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:14.772854 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:14.772858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:14.772881 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:14.773192 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.773200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.774164 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.774170 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.775474 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:14.775486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.776549 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.776556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:14.776734 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.776771 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 14:36:14.776896 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 14:36:14.777015 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":392,"threads_started":1}
I20260810 14:36:14.777505 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.771941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:52777 (local address 127.6.46.194:42227)
0810 14:36:14.772122 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:14.772127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:14.772532 (+ 405us) server_negotiation.cc:407] Connection header received
0810 14:36:14.772697 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:14.772702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:14.772756 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:14.772828 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:14.773342 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.773999 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.775649 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.776392 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.776530 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.776596 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:14.777161 (+ 565us) server_negotiation.cc:299] Negotiation successful
0810 14:36:14.777293 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:36:14.777555 3900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42efd5185894d9b8d93a9e62dbd35c2" candidate_uuid: "983642e8c1ab4c4c8985b9c5f433b616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f612521b56714c43b8041767379cef94" is_pre_election: true
I20260810 14:36:14.778083 3900 raft_consensus.cc:2454] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 983642e8c1ab4c4c8985b9c5f433b616 in term 0.
I20260810 14:36:14.778362 4104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42efd5185894d9b8d93a9e62dbd35c2" candidate_uuid: "983642e8c1ab4c4c8985b9c5f433b616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a7b542ea19461db1d56e00cc76191f" is_pre_election: true
I20260810 14:36:14.778662 4211 leader_election.cc:302] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 983642e8c1ab4c4c8985b9c5f433b616, f612521b56714c43b8041767379cef94; no voters:
I20260810 14:36:14.778784 4104 raft_consensus.cc:2454] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 983642e8c1ab4c4c8985b9c5f433b616 in term 0.
I20260810 14:36:14.778990 4434 raft_consensus.cc:2781] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:36:14.779106 4434 raft_consensus.cc:482] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:36:14.779206 4434 raft_consensus.cc:3037] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:14.781277 4434 raft_consensus.cc:504] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7b542ea19461db1d56e00cc76191f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42227 } } peers { permanent_uuid: "983642e8c1ab4c4c8985b9c5f433b616" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 32965 } } peers { permanent_uuid: "f612521b56714c43b8041767379cef94" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 37091 } }
I20260810 14:36:14.781839 4434 leader_election.cc:288] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [CANDIDATE]: Term 1 election: Requested vote from peers 28a7b542ea19461db1d56e00cc76191f (127.6.46.194:42227), f612521b56714c43b8041767379cef94 (127.6.46.193:37091)
I20260810 14:36:14.782191 4104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42efd5185894d9b8d93a9e62dbd35c2" candidate_uuid: "983642e8c1ab4c4c8985b9c5f433b616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28a7b542ea19461db1d56e00cc76191f"
I20260810 14:36:14.782326 3900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c42efd5185894d9b8d93a9e62dbd35c2" candidate_uuid: "983642e8c1ab4c4c8985b9c5f433b616" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f612521b56714c43b8041767379cef94"
I20260810 14:36:14.782389 4104 raft_consensus.cc:3037] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:14.782510 3900 raft_consensus.cc:3037] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:14.784543 3900 raft_consensus.cc:2454] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983642e8c1ab4c4c8985b9c5f433b616 in term 1.
I20260810 14:36:14.784713 4104 raft_consensus.cc:2454] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 983642e8c1ab4c4c8985b9c5f433b616 in term 1.
I20260810 14:36:14.784878 4211 leader_election.cc:302] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 983642e8c1ab4c4c8985b9c5f433b616, f612521b56714c43b8041767379cef94; no voters:
I20260810 14:36:14.785159 4434 raft_consensus.cc:2781] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:14.785539 4434 raft_consensus.cc:686] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [term 1 LEADER]: Becoming Leader. State: Replica: 983642e8c1ab4c4c8985b9c5f433b616, State: Running, Role: LEADER
I20260810 14:36:14.785970 4434 consensus_queue.cc:237] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7b542ea19461db1d56e00cc76191f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42227 } } peers { permanent_uuid: "983642e8c1ab4c4c8985b9c5f433b616" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 32965 } } peers { permanent_uuid: "f612521b56714c43b8041767379cef94" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 37091 } }
I20260810 14:36:14.790272 3654 catalog_manager.cc:5495] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 reported cstate change: term changed from 0 to 1, leader changed from <none> to 983642e8c1ab4c4c8985b9c5f433b616 (127.6.46.195). New cstate: current_term: 1 leader_uuid: "983642e8c1ab4c4c8985b9c5f433b616" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28a7b542ea19461db1d56e00cc76191f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "983642e8c1ab4c4c8985b9c5f433b616" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 32965 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f612521b56714c43b8041767379cef94" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 37091 } health_report { overall_health: UNKNOWN } } }
I20260810 14:36:14.814634 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.809773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34864 (local address 127.6.46.195:32965)
0810 14:36:14.809933 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:14.809937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:14.810154 (+ 217us) server_negotiation.cc:407] Connection header received
0810 14:36:14.810325 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:14.810328 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:14.810404 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:14.810511 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:36:14.810825 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.811587 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.812915 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.813310 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.813399 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.813789 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:36:14.814243 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:36:14.814343 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:14.814410 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 14:36:14.814468 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:36:14.821874 3900 raft_consensus.cc:1261] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94 [term 1 FOLLOWER]: Refusing update from remote peer 983642e8c1ab4c4c8985b9c5f433b616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:14.821908 4104 raft_consensus.cc:1261] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f [term 1 FOLLOWER]: Refusing update from remote peer 983642e8c1ab4c4c8985b9c5f433b616: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:14.822693 4439 consensus_queue.cc:1035] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28a7b542ea19461db1d56e00cc76191f" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 42227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:14.823022 4434 consensus_queue.cc:1035] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f612521b56714c43b8041767379cef94" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 37091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:14.838825 4444 mvcc.cc:205] Tried to move back new op lower bound from 7316982066461618176 to 7316982066327113728. Current Snapshot: MvccSnapshot[applied={T|T < 7316982066461618176}]
I20260810 14:36:14.848033 3656 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:14.848284 3656 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:14.851845 3656 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7667e3074634484aeac393b876dc231: Sending DeleteTablet for 3 replicas of tablet c42efd5185894d9b8d93a9e62dbd35c2
I20260810 14:36:14.852960 4075 tablet_service.cc:1514] Processing DeleteTablet for tablet c42efd5185894d9b8d93a9e62dbd35c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:14 UTC) from {username='slave'} at 127.0.0.1:58206
I20260810 14:36:14.853008 4293 tablet_service.cc:1514] Processing DeleteTablet for tablet c42efd5185894d9b8d93a9e62dbd35c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:14 UTC) from {username='slave'} at 127.0.0.1:34860
I20260810 14:36:14.853616 3865 tablet_service.cc:1514] Processing DeleteTablet for tablet c42efd5185894d9b8d93a9e62dbd35c2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:14 UTC) from {username='slave'} at 127.0.0.1:40500
I20260810 14:36:14.853814 4462 tablet_replica.cc:330] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f: stopping tablet replica
I20260810 14:36:14.854223 4462 raft_consensus.cc:2229] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:14.854425 4466 tablet_replica.cc:330] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94: stopping tablet replica
I20260810 14:36:14.854784 4466 raft_consensus.cc:2229] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:14.855249 4466 raft_consensus.cc:2258] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:14.855265 4462 raft_consensus.cc:2258] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:14.855463 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 3762
I20260810 14:36:14.858080 4462 ts_tablet_manager.cc:1905] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:14.863385 4466 ts_tablet_manager.cc:1905] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:14.865332 4462 ts_tablet_manager.cc:1918] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:14.865486 4462 log.cc:1196] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/c42efd5185894d9b8d93a9e62dbd35c2
I20260810 14:36:14.866086 4462 ts_tablet_manager.cc:1939] T c42efd5185894d9b8d93a9e62dbd35c2 P 28a7b542ea19461db1d56e00cc76191f: Deleting consensus metadata
I20260810 14:36:14.867049 4465 tablet_replica.cc:330] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616: stopping tablet replica
I20260810 14:36:14.867555 4465 raft_consensus.cc:2229] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:36:14.867808 3635 catalog_manager.cc:4841] TS 28a7b542ea19461db1d56e00cc76191f (127.6.46.194:42227): tablet c42efd5185894d9b8d93a9e62dbd35c2 (table test-table [id=80d2ce1ed49c490c8074de76959836af]) successfully deleted
I20260810 14:36:14.868427 4465 raft_consensus.cc:2258] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:14.870208 4466 ts_tablet_manager.cc:1918] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:14.870390 4466 log.cc:1196] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/c42efd5185894d9b8d93a9e62dbd35c2
I20260810 14:36:14.870929 4466 ts_tablet_manager.cc:1939] T c42efd5185894d9b8d93a9e62dbd35c2 P f612521b56714c43b8041767379cef94: Deleting consensus metadata
I20260810 14:36:14.871184 4465 ts_tablet_manager.cc:1905] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:14.872500 3633 catalog_manager.cc:4841] TS f612521b56714c43b8041767379cef94 (127.6.46.193:37091): tablet c42efd5185894d9b8d93a9e62dbd35c2 (table test-table [id=80d2ce1ed49c490c8074de76959836af]) successfully deleted
I20260810 14:36:14.876400 4426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:14.856177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.6.46.193:37091)
0810 14:36:14.856338 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:14.856344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:14.868897 (+ 12553us) server_negotiation.cc:407] Connection header received
0810 14:36:14.869127 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:14.869132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:14.869224 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:14.869310 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:14.869766 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.870880 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.872791 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:14.873398 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:14.873490 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:14.874884 (+ 1394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:14.874894 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:14.874910 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:14.875589 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:14.875644 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:14.875668 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:14.875783 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:14.875939 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 14:36:14.876047 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:36:14.877049 3968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:14.877944 4465 ts_tablet_manager.cc:1918] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:14.878180 4465 log.cc:1196] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/c42efd5185894d9b8d93a9e62dbd35c2
I20260810 14:36:14.878729 4465 ts_tablet_manager.cc:1939] T c42efd5185894d9b8d93a9e62dbd35c2 P 983642e8c1ab4c4c8985b9c5f433b616: Deleting consensus metadata
I20260810 14:36:14.880438 3634 catalog_manager.cc:4841] TS 983642e8c1ab4c4c8985b9c5f433b616 (127.6.46.195:32965): tablet c42efd5185894d9b8d93a9e62dbd35c2 (table test-table [id=80d2ce1ed49c490c8074de76959836af]) successfully deleted
I20260810 14:36:14.999082 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 3762
I20260810 14:36:15.029608 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 3979
I20260810 14:36:15.037132 4419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:15.030491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58210 (local address 127.6.46.194:42227)
0810 14:36:15.030654 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:15.030660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:15.030678 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:36:15.030844 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:15.030848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:15.030948 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:15.031032 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:15.031481 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.032184 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.034165 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.034583 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.034696 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:15.035827 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:15.035837 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:15.035854 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:15.036443 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:15.036490 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:15.036512 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:15.036597 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:15.036736 (+ 139us) server_negotiation.cc:299] Negotiation successful
0810 14:36:15.036830 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:36:15.037817 4170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:15.125092 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 3979
I20260810 14:36:15.147100 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 4186
I20260810 14:36:15.154963 4425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:15.147985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34880 (local address 127.6.46.195:32965)
0810 14:36:15.148159 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:15.148164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:15.148239 (+ 75us) server_negotiation.cc:407] Connection header received
0810 14:36:15.148439 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:15.148443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:15.148522 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:15.148610 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:15.149111 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.149879 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.151720 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:15.152183 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:15.152280 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:15.153477 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:15.153486 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:15.153501 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:15.154227 (+ 726us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:15.154276 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:15.154296 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:15.154379 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:15.154547 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 14:36:15.154625 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 14:36:15.155725 4380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:15.274109 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 4186
I20260810 14:36:15.297238 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 3605
I20260810 14:36:15.298310 3707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:15.404196 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 3605
2026-08-10T14:36:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2649 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:15 dist-test-slave-q9h9 krb5kdc[4477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:15 dist-test-slave-q9h9 krb5kdc[4477](info): set up 2 sockets
Aug 10 14:36:15 dist-test-slave-q9h9 krb5kdc[4477](info): commencing operation
krb5kdc: starting...
W20260810 14:36:16.638192 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:16 dist-test-slave-q9h9 krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:36:16.896719 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:33227
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:33227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:17.216372 4502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:17.216641 4502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:17.216755 4502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:17.223665 4502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:17.223757 4502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:17.223825 4502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:17.223909 4502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:17.223961 4502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:17.233295 4502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:33227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:33227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:17.234499 4502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:17.235755 4502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:17.246853 4511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:17.246894 4515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:17.246923 4512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:17.247664 4502 server_base.cc:1034] running on GCE node
I20260810 14:36:17.248394 4502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:17.249691 4502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:17.250890 4502 hybrid_clock.cc:648] HybridClock initialized: now 1786372577250836 us; error 41 us; skew 500 ppm
I20260810 14:36:17.251225 4502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:17 dist-test-slave-q9h9 krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372577, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:17.258925 4502 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:36:17.260288 4502 webserver.cc:460] Webserver started at http://127.6.46.254:44383/ using document root <none> and password file <none>
I20260810 14:36:17.260932 4502 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:17.261057 4502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:17.261343 4502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:17.264220 4502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "34b0b2552cb54dc196d4bb9e1f423a81"
format_stamp: "Formatted at 2026-08-10 14:36:17 on dist-test-slave-q9h9"
server_key: "acd2ba716e8e1f3e4d3a1f515dcf5cfe"
server_key_iv: "df5be6aa5d21d8c3e8b721cacbd2f24d"
server_key_version: "encryptionkey@0"
I20260810 14:36:17.264938 4502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "34b0b2552cb54dc196d4bb9e1f423a81"
format_stamp: "Formatted at 2026-08-10 14:36:17 on dist-test-slave-q9h9"
server_key: "acd2ba716e8e1f3e4d3a1f515dcf5cfe"
server_key_iv: "df5be6aa5d21d8c3e8b721cacbd2f24d"
server_key_version: "encryptionkey@0"
I20260810 14:36:17.269762 4502 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260810 14:36:17.272984 4524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:17.274257 4502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:17.274410 4502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "34b0b2552cb54dc196d4bb9e1f423a81"
format_stamp: "Formatted at 2026-08-10 14:36:17 on dist-test-slave-q9h9"
server_key: "acd2ba716e8e1f3e4d3a1f515dcf5cfe"
server_key_iv: "df5be6aa5d21d8c3e8b721cacbd2f24d"
server_key_version: "encryptionkey@0"
I20260810 14:36:17.274585 4502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:17.299728 4502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:17.303370 4502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:17.303632 4502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:17.320585 4502 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:33227
I20260810 14:36:17.320617 4602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:33227 every 8 connection(s)
I20260810 14:36:17.321775 4502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:17.325224 4604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:17.329703 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 4502
I20260810 14:36:17.329893 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:17.330518 6331 external_mini_cluster.cc:1428] Setting key 86f8905b44a435146710357b77e576d4
I20260810 14:36:17.336627 4604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81: Bootstrap starting.
I20260810 14:36:17.340035 4604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:17.341217 4604 log.cc:824] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:17.344334 4604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81: No bootstrap required, opened a new log
Aug 10 14:36:17 dist-test-slave-q9h9 krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:36:17.354247 4604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b0b2552cb54dc196d4bb9e1f423a81" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33227 } }
I20260810 14:36:17.354666 4604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:17.354698 4608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:17.332349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.6.46.254:33227)
0810 14:36:17.333341 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:17.333355 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:17.333408 (+ 53us) server_negotiation.cc:407] Connection header received
0810 14:36:17.335299 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:17.335319 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:17.335669 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:17.335954 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:17.337804 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.339233 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.341055 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.341832 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.342031 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:17.346553 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:17.346572 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:17.346595 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:17.346636 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:17.350444 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:17.351032 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:17.351044 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:17.351052 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:17.351266 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:17.351604 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:17.351608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:17.351611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:17.352301 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:17.352631 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:17.352989 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 14:36:17.353389 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":452,"threads_started":1}
I20260810 14:36:17.354807 4604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34b0b2552cb54dc196d4bb9e1f423a81, State: Initialized, Role: FOLLOWER
I20260810 14:36:17.355427 4604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b0b2552cb54dc196d4bb9e1f423a81" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33227 } }
I20260810 14:36:17.355706 4604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:17.355847 4604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:17.356043 4604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:17.358435 4604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b0b2552cb54dc196d4bb9e1f423a81" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33227 } }
I20260810 14:36:17.358878 4604 leader_election.cc:302] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34b0b2552cb54dc196d4bb9e1f423a81; no voters:
I20260810 14:36:17.359392 4604 leader_election.cc:288] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:17.359508 4610 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:17.360153 4610 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [term 1 LEADER]: Becoming Leader. State: Replica: 34b0b2552cb54dc196d4bb9e1f423a81, State: Running, Role: LEADER
I20260810 14:36:17.360661 4610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b0b2552cb54dc196d4bb9e1f423a81" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33227 } }
I20260810 14:36:17.361316 4604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:17.363538 4614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34b0b2552cb54dc196d4bb9e1f423a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b0b2552cb54dc196d4bb9e1f423a81" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33227 } } }
I20260810 14:36:17.363564 4615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34b0b2552cb54dc196d4bb9e1f423a81. Latest consensus state: current_term: 1 leader_uuid: "34b0b2552cb54dc196d4bb9e1f423a81" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34b0b2552cb54dc196d4bb9e1f423a81" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33227 } } }
I20260810 14:36:17.363966 4614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:17.363977 4615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:17.364759 4625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:17.369127 4625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:17.376941 4625 catalog_manager.cc:1347] Generated new cluster ID: ffbf05a4135342d48811ba5a606fadb3
I20260810 14:36:17.377040 4625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:17.406407 4625 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:17.408165 4625 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:17.420715 4625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81: Generated new TSK 0
I20260810 14:36:17.421646 4625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:36:17.483661 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:33227
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:17.806105 4651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:17.806357 4651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:17.806480 4651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:17.813333 4651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:17.813426 4651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:17.813570 4651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:36:17.822659 4651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:17.823730 4651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:17.824900 4651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:17.836964 4667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:17.837015 4666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:17.837042 4671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:17.837971 4651 server_base.cc:1034] running on GCE node
I20260810 14:36:17.838548 4651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:17.839391 4651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:17.840605 4651 hybrid_clock.cc:648] HybridClock initialized: now 1786372577840561 us; error 38 us; skew 500 ppm
I20260810 14:36:17.840885 4651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:17 dist-test-slave-q9h9 krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372577, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:17.843904 4651 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:36:17.845207 4651 webserver.cc:460] Webserver started at http://127.6.46.193:40561/ using document root <none> and password file <none>
I20260810 14:36:17.845810 4651 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:17.845927 4651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:17.846252 4651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:17.848958 4651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "78b180891c294398a87c88625f4999de"
format_stamp: "Formatted at 2026-08-10 14:36:17 on dist-test-slave-q9h9"
server_key: "e352c3c17868f0daf172d3be6513aed6"
server_key_iv: "8076d0cdef57b71584fe048b943449f2"
server_key_version: "encryptionkey@0"
I20260810 14:36:17.849696 4651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "78b180891c294398a87c88625f4999de"
format_stamp: "Formatted at 2026-08-10 14:36:17 on dist-test-slave-q9h9"
server_key: "e352c3c17868f0daf172d3be6513aed6"
server_key_iv: "8076d0cdef57b71584fe048b943449f2"
server_key_version: "encryptionkey@0"
I20260810 14:36:17.854614 4651 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260810 14:36:17.858088 4684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:17.859246 4651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260810 14:36:17.859391 4651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "78b180891c294398a87c88625f4999de"
format_stamp: "Formatted at 2026-08-10 14:36:17 on dist-test-slave-q9h9"
server_key: "e352c3c17868f0daf172d3be6513aed6"
server_key_iv: "8076d0cdef57b71584fe048b943449f2"
server_key_version: "encryptionkey@0"
I20260810 14:36:17.859562 4651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:17.878052 4651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:17.881666 4651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:17.881906 4651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:17.882800 4651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:17.884043 4651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:17.884128 4651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:17.884251 4651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:17.884303 4651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:17.914089 4651 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:44311
I20260810 14:36:17.914160 4868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:44311 every 8 connection(s)
I20260810 14:36:17.915302 4651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:36:17.917780 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 4651
I20260810 14:36:17.917972 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:36:17.918448 6331 external_mini_cluster.cc:1428] Setting key c978e9eb5242daf0db58f9944f3984fc
Aug 10 14:36:17 dist-test-slave-q9h9 krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372577, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:36:17.934098 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:17.917605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:52097 (local address 127.6.46.254:33227)
0810 14:36:17.918147 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:17.918153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:17.919100 (+ 947us) server_negotiation.cc:407] Connection header received
0810 14:36:17.921266 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:17.921271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:17.921347 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:17.921501 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:17.922982 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.923738 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.925187 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.925584 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.925691 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:17.928862 (+ 3171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:17.928867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:17.928870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:17.928901 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:17.931216 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:17.931873 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:17.931877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:17.931879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:17.931998 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:17.932392 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:17.932395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:17.932397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:17.932654 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:17.932784 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:17.933488 (+ 704us) server_negotiation.cc:299] Negotiation successful
0810 14:36:17.933712 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":235,"threads_started":1}
I20260810 14:36:17.934927 4877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:17.918152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:33227 (local address 127.6.46.193:52097)
0810 14:36:17.918917 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:17.918971 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:17.921021 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:17.921740 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:17.921748 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:17.922278 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:17.922824 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:17.922841 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.923885 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.923889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:17.925055 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:17.925067 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:17.925791 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:17.925798 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:17.926004 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:17.926638 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:17.926662 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:17.928669 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:17.931339 (+ 2670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:17.931348 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:17.931362 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:17.931754 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:17.932102 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:17.932107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:17.932111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:17.932282 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:17.932774 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:17.932781 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:17.933126 (+ 345us) client_negotiation.cc:769] Sending connection context
0810 14:36:17.933344 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 14:36:17.933817 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":310,"threads_started":1}
I20260810 14:36:17.936378 4871 heartbeater.cc:347] Connected to a master server at 127.6.46.254:33227
I20260810 14:36:17.936717 4871 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:17.937505 4871 heartbeater.cc:511] Master 127.6.46.254:33227 requested a full tablet report, sending...
I20260810 14:36:17.939911 4547 ts_manager.cc:194] Registered new tserver with Master: 78b180891c294398a87c88625f4999de (127.6.46.193:44311)
I20260810 14:36:17.941782 4547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:52097
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:36:17.981642 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:33227
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:18.293848 4884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:18.294129 4884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:18.294220 4884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:18.301137 4884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:18.301230 4884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:18.301354 4884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:36:18.310534 4884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:18.311640 4884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:18.312819 4884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:18.326413 4895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:18.326386 4896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:18.327258 4900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:18.327847 4884 server_base.cc:1034] running on GCE node
I20260810 14:36:18.328433 4884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:18.329330 4884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:18.330535 4884 hybrid_clock.cc:648] HybridClock initialized: now 1786372578330491 us; error 35 us; skew 500 ppm
I20260810 14:36:18.330816 4884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:18 dist-test-slave-q9h9 krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372578, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:18.333910 4884 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:36:18.335330 4884 webserver.cc:460] Webserver started at http://127.6.46.194:39577/ using document root <none> and password file <none>
I20260810 14:36:18.335949 4884 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:18.336064 4884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:18.336350 4884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:18.339085 4884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "1b336ef0d9f4464ca4a69aad84c3c562"
format_stamp: "Formatted at 2026-08-10 14:36:18 on dist-test-slave-q9h9"
server_key: "a0f425cdc1ac51ee96a90d4d4f550521"
server_key_iv: "e7e0c2ce0382a2cf235004259d108a52"
server_key_version: "encryptionkey@0"
I20260810 14:36:18.339818 4884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "1b336ef0d9f4464ca4a69aad84c3c562"
format_stamp: "Formatted at 2026-08-10 14:36:18 on dist-test-slave-q9h9"
server_key: "a0f425cdc1ac51ee96a90d4d4f550521"
server_key_iv: "e7e0c2ce0382a2cf235004259d108a52"
server_key_version: "encryptionkey@0"
I20260810 14:36:18.344631 4884 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 14:36:18.347880 4919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:18.349092 4884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:36:18.349239 4884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "1b336ef0d9f4464ca4a69aad84c3c562"
format_stamp: "Formatted at 2026-08-10 14:36:18 on dist-test-slave-q9h9"
server_key: "a0f425cdc1ac51ee96a90d4d4f550521"
server_key_iv: "e7e0c2ce0382a2cf235004259d108a52"
server_key_version: "encryptionkey@0"
I20260810 14:36:18.349411 4884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:18.360453 4884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:18.364333 4884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:18.364584 4884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:18.365435 4884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:18.366779 4884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:18.366856 4884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:18.366966 4884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:18.367012 4884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:18.394507 4884 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:40817
I20260810 14:36:18.394556 5098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:40817 every 8 connection(s)
I20260810 14:36:18.395737 4884 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:36:18.403717 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 4884
I20260810 14:36:18.403882 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:36:18.404340 6331 external_mini_cluster.cc:1428] Setting key 8ade0fe7eb867bc4bc832767657f2f0b
Aug 10 14:36:18 dist-test-slave-q9h9 krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372578, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:36:18.415061 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.398183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:38479 (local address 127.6.46.254:33227)
0810 14:36:18.398357 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:18.398362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:18.399436 (+ 1074us) server_negotiation.cc:407] Connection header received
0810 14:36:18.401173 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:18.401178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:18.401247 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:18.401335 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:18.402753 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.403574 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.405136 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.405642 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.405733 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.409127 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:18.409133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:18.409138 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:18.409175 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:18.412175 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:18.412859 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:18.412864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:18.412866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:18.413017 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:18.413440 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:18.413444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:18.413447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:18.413755 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:18.413887 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:18.414517 (+ 630us) server_negotiation.cc:299] Negotiation successful
0810 14:36:18.414754 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:36:18.415758 5106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.398585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:33227 (local address 127.6.46.194:38479)
0810 14:36:18.399276 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:18.399313 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:18.400897 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:18.401515 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:18.401523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:18.401999 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:18.402591 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.402609 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.403731 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.403735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.405018 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.405030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.405866 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.405873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.406117 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.406706 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:18.406734 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:18.408903 (+ 2169us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:18.412317 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:18.412324 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:18.412335 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:18.412736 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:18.413142 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:18.413146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:18.413149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:18.413328 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:18.413872 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:18.413877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:18.414177 (+ 300us) client_negotiation.cc:769] Sending connection context
0810 14:36:18.414386 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 14:36:18.414736 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":271,"threads_started":1}
I20260810 14:36:18.417114 5102 heartbeater.cc:347] Connected to a master server at 127.6.46.254:33227
I20260810 14:36:18.417373 5102 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:18.418040 5102 heartbeater.cc:511] Master 127.6.46.254:33227 requested a full tablet report, sending...
I20260810 14:36:18.419359 4547 ts_manager.cc:194] Registered new tserver with Master: 1b336ef0d9f4464ca4a69aad84c3c562 (127.6.46.194:40817)
I20260810 14:36:18.420127 4547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:38479
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:36:18.462829 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:33227
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:33329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:18.786091 5111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:18.786365 5111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:18.786505 5111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:18.793427 5111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:18.793534 5111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:18.793691 5111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:36:18.803027 5111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:33227
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:18.804172 5111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:18.805383 5111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:18.817391 5119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:18.817449 5129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:18.817430 5124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:18.818435 5111 server_base.cc:1034] running on GCE node
I20260810 14:36:18.818972 5111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:18.819788 5111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:18.820978 5111 hybrid_clock.cc:648] HybridClock initialized: now 1786372578820938 us; error 30 us; skew 500 ppm
I20260810 14:36:18.821255 5111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:18 dist-test-slave-q9h9 krb5kdc[4477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372578, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:18.824148 5111 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:36:18.825472 5111 webserver.cc:460] Webserver started at http://127.6.46.195:38657/ using document root <none> and password file <none>
I20260810 14:36:18.826104 5111 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:18.826224 5111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:18.826514 5111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:18.829140 5111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "c215b4e9937d4c75acc517f322cf3175"
format_stamp: "Formatted at 2026-08-10 14:36:18 on dist-test-slave-q9h9"
server_key: "eef733b0e70b62ff96f61871e849f518"
server_key_iv: "5da769f2b799e54f7164dca054f74dc7"
server_key_version: "encryptionkey@0"
I20260810 14:36:18.829852 5111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "c215b4e9937d4c75acc517f322cf3175"
format_stamp: "Formatted at 2026-08-10 14:36:18 on dist-test-slave-q9h9"
server_key: "eef733b0e70b62ff96f61871e849f518"
server_key_iv: "5da769f2b799e54f7164dca054f74dc7"
server_key_version: "encryptionkey@0"
I20260810 14:36:18.834673 5111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 14:36:18.837771 5141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:18.838950 5111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:36:18.839098 5111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "c215b4e9937d4c75acc517f322cf3175"
format_stamp: "Formatted at 2026-08-10 14:36:18 on dist-test-slave-q9h9"
server_key: "eef733b0e70b62ff96f61871e849f518"
server_key_iv: "5da769f2b799e54f7164dca054f74dc7"
server_key_version: "encryptionkey@0"
I20260810 14:36:18.839265 5111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:18.854295 5111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:18.858042 5111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:18.858289 5111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:18.859104 5111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:18.860276 5111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:18.860353 5111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:18.860467 5111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:18.860529 5111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:18.889075 5111 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:44751
I20260810 14:36:18.889129 5329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:44751 every 8 connection(s)
I20260810 14:36:18.890250 5111 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:36:18.895599 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 5111
I20260810 14:36:18.895761 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:36:18.896188 6331 external_mini_cluster.cc:1428] Setting key c4dd199acd2148d5bcdc325bc263df32
Aug 10 14:36:18 dist-test-slave-q9h9 krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372578, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:36:18.907963 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.892500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:53643 (local address 127.6.46.254:33227)
0810 14:36:18.892671 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:18.892676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:18.893765 (+ 1089us) server_negotiation.cc:407] Connection header received
0810 14:36:18.895588 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:18.895593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:18.895674 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:18.895774 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:18.897303 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.898270 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.899859 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.900252 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.900343 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.903307 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:18.903311 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:18.903315 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:18.903348 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:18.905551 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:18.906212 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:18.906216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:18.906219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:18.906332 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:18.906671 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:18.906674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:18.906676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:18.906917 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:18.907024 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:18.907546 (+ 522us) server_negotiation.cc:299] Negotiation successful
0810 14:36:18.907722 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:36:18.908733 5332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.892942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:33227 (local address 127.6.46.195:53643)
0810 14:36:18.893625 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:18.893660 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:18.895333 (+ 1673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:18.895997 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:18.896013 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:18.896511 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:18.897132 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.897152 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.898442 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.898446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.899720 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.899733 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.900414 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.900419 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.900603 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.901133 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:18.901154 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:18.903119 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:18.905671 (+ 2552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:18.905677 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:18.905687 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:18.906127 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:18.906409 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:18.906412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:18.906414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:18.906586 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:18.907003 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:18.907007 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:18.907268 (+ 261us) client_negotiation.cc:769] Sending connection context
0810 14:36:18.907469 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 14:36:18.907779 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":274,"threads_started":1}
I20260810 14:36:18.909981 5330 heartbeater.cc:347] Connected to a master server at 127.6.46.254:33227
I20260810 14:36:18.910270 5330 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:18.910996 5330 heartbeater.cc:511] Master 127.6.46.254:33227 requested a full tablet report, sending...
I20260810 14:36:18.912272 4547 ts_manager.cc:194] Registered new tserver with Master: c215b4e9937d4c75acc517f322cf3175 (127.6.46.195:44751)
I20260810 14:36:18.913009 4547 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:53643
I20260810 14:36:18.922878 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:36:18.938227 4874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.926630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.6.46.254:33227)
0810 14:36:18.926807 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:18.926813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:18.927194 (+ 381us) server_negotiation.cc:407] Connection header received
0810 14:36:18.927310 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:18.927313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:18.927366 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:18.927439 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:18.928725 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.929394 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.930671 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.931048 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.931132 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.933507 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:18.933511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:18.933515 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:18.933543 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:18.935972 (+ 2429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:18.936606 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:18.936609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:18.936611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:18.936728 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:18.937043 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:18.937046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:18.937048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:18.937313 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:18.937496 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:18.937772 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 14:36:18.937894 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 14:36:18.945101 4871 heartbeater.cc:503] Master 127.6.46.254:33227 was elected leader, sending a full tablet report...
I20260810 14:36:18.941545 4547 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:36:18.948794 4547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:36:18.979684 5355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.960541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:40817 (local address 127.0.0.1:33872)
0810 14:36:18.970833 (+ 10292us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:18.970850 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:18.970959 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:18.971884 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:18.971894 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:18.971924 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:18.972425 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.972438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.974507 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.974524 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.976825 (+ 2301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.976838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.978708 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.978714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.978901 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.978958 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 14:36:18.979087 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 14:36:18.979262 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10168,"thread_start_us":8834,"threads_started":1}
I20260810 14:36:18.979773 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.960427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:44751 (local address 127.0.0.1:44976)
0810 14:36:18.969726 (+ 9299us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:18.969763 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:18.969951 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:18.971736 (+ 1785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:18.971745 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:18.971779 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:18.972280 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.972297 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.974497 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.974514 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.977093 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.977110 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.978954 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.978961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.979141 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.979173 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 14:36:18.979250 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 14:36:18.979367 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9068,"thread_start_us":10149,"threads_started":1}
I20260810 14:36:18.979929 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.961066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33872 (local address 127.6.46.194:40817)
0810 14:36:18.970421 (+ 9355us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:18.970432 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:18.971086 (+ 654us) server_negotiation.cc:407] Connection header received
0810 14:36:18.971208 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:18.971217 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:18.971437 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:18.971641 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:18.972643 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.974311 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.976981 (+ 2670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.978477 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.979202 (+ 725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.979382 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:18.979459 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 14:36:18.979627 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9188,"thread_start_us":9034,"threads_started":1}
I20260810 14:36:18.980438 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.960668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.6.46.195:44751)
0810 14:36:18.970792 (+ 10124us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:18.970800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:18.970830 (+ 30us) server_negotiation.cc:407] Connection header received
0810 14:36:18.970932 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:18.970940 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:18.971192 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:18.971390 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:18.972542 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.974321 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.977277 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.978439 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.978569 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.978809 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:18.979937 (+ 1128us) server_negotiation.cc:299] Negotiation successful
0810 14:36:18.980102 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9966,"thread_start_us":9786,"threads_started":1}
I20260810 14:36:18.983690 5002 tablet_service.cc:1467] Processing CreateTablet for tablet c3d41b8f6784405c9983deab6a8d08f2 (DEFAULT_TABLE table=test-table [id=fed44077db4b44c6b7c56d001373900b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:18.983790 5222 tablet_service.cc:1467] Processing CreateTablet for tablet c3d41b8f6784405c9983deab6a8d08f2 (DEFAULT_TABLE table=test-table [id=fed44077db4b44c6b7c56d001373900b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:18.984922 5002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d41b8f6784405c9983deab6a8d08f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:18.985204 5222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d41b8f6784405c9983deab6a8d08f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:18.985728 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.970944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:44311 (local address 127.0.0.1:57840)
0810 14:36:18.975008 (+ 4064us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:18.975028 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:18.975127 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:18.978118 (+ 2991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:18.978123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:18.978149 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:18.978590 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.978603 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.981259 (+ 2656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.981264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.983202 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:18.983216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.984958 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.984965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:18.985130 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.985162 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 14:36:18.985252 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 14:36:18.985370 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3950,"mutex_wait_us":3,"thread_start_us":3787,"threads_started":1}
I20260810 14:36:18.985752 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:18.971191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57840 (local address 127.6.46.193:44311)
0810 14:36:18.977479 (+ 6288us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:18.977487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:18.977523 (+ 36us) server_negotiation.cc:407] Connection header received
0810 14:36:18.977608 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:18.977616 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:18.977852 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:18.978814 (+ 962us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:18.979049 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.981116 (+ 2067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.983367 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:18.984743 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:18.984889 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:18.985122 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:18.985294 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 14:36:18.985454 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6120,"thread_start_us":5912,"threads_started":1}
I20260810 14:36:18.993453 4769 tablet_service.cc:1467] Processing CreateTablet for tablet c3d41b8f6784405c9983deab6a8d08f2 (DEFAULT_TABLE table=test-table [id=fed44077db4b44c6b7c56d001373900b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:18.994874 4769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3d41b8f6784405c9983deab6a8d08f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:18.996227 5365 tablet_bootstrap.cc:492] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175: Bootstrap starting.
I20260810 14:36:18.998915 5366 tablet_bootstrap.cc:492] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562: Bootstrap starting.
I20260810 14:36:19.000222 5365 tablet_bootstrap.cc:654] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:19.001485 5365 log.cc:824] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:19.002388 5366 tablet_bootstrap.cc:654] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:19.003453 5366 log.cc:824] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:19.004730 5365 tablet_bootstrap.cc:492] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175: No bootstrap required, opened a new log
I20260810 14:36:19.005062 5365 ts_tablet_manager.cc:1397] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260810 14:36:19.006361 5366 tablet_bootstrap.cc:492] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562: No bootstrap required, opened a new log
I20260810 14:36:19.006619 5366 ts_tablet_manager.cc:1397] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260810 14:36:19.007354 5369 tablet_bootstrap.cc:492] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de: Bootstrap starting.
I20260810 14:36:19.010831 5369 tablet_bootstrap.cc:654] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:19.011993 5369 log.cc:824] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:19.014796 5365 raft_consensus.cc:348] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c215b4e9937d4c75acc517f322cf3175" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44751 } } peers { permanent_uuid: "1b336ef0d9f4464ca4a69aad84c3c562" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 40817 } } peers { permanent_uuid: "78b180891c294398a87c88625f4999de" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44311 } }
I20260810 14:36:19.014688 5366 raft_consensus.cc:348] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c215b4e9937d4c75acc517f322cf3175" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44751 } } peers { permanent_uuid: "1b336ef0d9f4464ca4a69aad84c3c562" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 40817 } } peers { permanent_uuid: "78b180891c294398a87c88625f4999de" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44311 } }
I20260810 14:36:19.015132 5365 raft_consensus.cc:374] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:19.015137 5366 raft_consensus.cc:374] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:19.015237 5369 tablet_bootstrap.cc:492] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de: No bootstrap required, opened a new log
I20260810 14:36:19.015247 5365 raft_consensus.cc:729] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c215b4e9937d4c75acc517f322cf3175, State: Initialized, Role: FOLLOWER
I20260810 14:36:19.015275 5366 raft_consensus.cc:729] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b336ef0d9f4464ca4a69aad84c3c562, State: Initialized, Role: FOLLOWER
I20260810 14:36:19.015522 5369 ts_tablet_manager.cc:1397] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260810 14:36:19.015829 5365 consensus_queue.cc:260] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c215b4e9937d4c75acc517f322cf3175" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44751 } } peers { permanent_uuid: "1b336ef0d9f4464ca4a69aad84c3c562" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 40817 } } peers { permanent_uuid: "78b180891c294398a87c88625f4999de" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44311 } }
I20260810 14:36:19.015901 5366 consensus_queue.cc:260] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c215b4e9937d4c75acc517f322cf3175" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44751 } } peers { permanent_uuid: "1b336ef0d9f4464ca4a69aad84c3c562" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 40817 } } peers { permanent_uuid: "78b180891c294398a87c88625f4999de" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44311 } }
I20260810 14:36:19.016840 5330 heartbeater.cc:503] Master 127.6.46.254:33227 was elected leader, sending a full tablet report...
I20260810 14:36:19.017449 5365 ts_tablet_manager.cc:1428] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20260810 14:36:19.018461 5102 heartbeater.cc:503] Master 127.6.46.254:33227 was elected leader, sending a full tablet report...
I20260810 14:36:19.019018 5366 ts_tablet_manager.cc:1428] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260810 14:36:19.025636 5369 raft_consensus.cc:348] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c215b4e9937d4c75acc517f322cf3175" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44751 } } peers { permanent_uuid: "1b336ef0d9f4464ca4a69aad84c3c562" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 40817 } } peers { permanent_uuid: "78b180891c294398a87c88625f4999de" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44311 } }
I20260810 14:36:19.025952 5369 raft_consensus.cc:374] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:19.026068 5369 raft_consensus.cc:729] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78b180891c294398a87c88625f4999de, State: Initialized, Role: FOLLOWER
I20260810 14:36:19.026572 5369 consensus_queue.cc:260] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c215b4e9937d4c75acc517f322cf3175" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44751 } } peers { permanent_uuid: "1b336ef0d9f4464ca4a69aad84c3c562" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 40817 } } peers { permanent_uuid: "78b180891c294398a87c88625f4999de" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44311 } }
I20260810 14:36:19.028019 5369 ts_tablet_manager.cc:1428] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de: Time spent starting tablet: real 0.012s user 0.005s sys 0.006s
W20260810 14:36:19.141714 5331 tablet.cc:2369] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:36:19.147622 5105 tablet.cc:2369] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:36:19.167415 4873 tablet.cc:2369] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:36:19.271999 5376 raft_consensus.cc:482] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:36:19.272320 5376 raft_consensus.cc:504] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c215b4e9937d4c75acc517f322cf3175" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44751 } } peers { permanent_uuid: "1b336ef0d9f4464ca4a69aad84c3c562" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 40817 } } peers { permanent_uuid: "78b180891c294398a87c88625f4999de" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44311 } }
I20260810 14:36:19.274091 5376 leader_election.cc:288] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c215b4e9937d4c75acc517f322cf3175 (127.6.46.195:44751), 1b336ef0d9f4464ca4a69aad84c3c562 (127.6.46.194:40817)
I20260810 14:36:19.282722 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.274997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:53689 (local address 127.6.46.194:40817)
0810 14:36:19.275202 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:19.275208 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:19.275376 (+ 168us) server_negotiation.cc:407] Connection header received
0810 14:36:19.275525 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:19.275530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:19.275600 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:19.275687 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:19.276327 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.277447 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.280377 (+ 2930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.281557 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.281725 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.281800 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:19.282320 (+ 520us) server_negotiation.cc:299] Negotiation successful
0810 14:36:19.282452 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 14:36:19.282747 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.274506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:53055 (local address 127.6.46.195:44751)
0810 14:36:19.274734 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:19.274740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:19.275175 (+ 435us) server_negotiation.cc:407] Connection header received
0810 14:36:19.275483 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:19.275490 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:19.275569 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:19.275685 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:19.276392 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.277460 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.280380 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.281567 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.281738 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.281802 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:19.282312 (+ 510us) server_negotiation.cc:299] Negotiation successful
0810 14:36:19.282452 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 14:36:19.282929 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.274271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:44751 (local address 127.6.46.193:53055)
0810 14:36:19.275043 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:19.275073 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:19.275301 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:19.275699 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:19.275704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:19.275806 (+ 102us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:19.276265 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:19.276277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.277598 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.277602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:19.280216 (+ 2614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:19.280233 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.281817 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.281822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:19.282107 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.282165 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 14:36:19.282253 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 14:36:19.282423 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":271,"spinlock_wait_cycles":24448,"thread_start_us":396,"threads_started":1}
I20260810 14:36:19.282955 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.274517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:40817 (local address 127.6.46.193:53689)
0810 14:36:19.275288 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:19.275304 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:19.275412 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:19.275699 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:19.275704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:19.275732 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:19.276159 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:19.276173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.277596 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.277602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:19.280228 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:19.280237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.281783 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.281790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:19.282103 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.282160 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 14:36:19.282251 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 14:36:19.282423 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":219,"spinlock_wait_cycles":18304,"thread_start_us":471,"threads_started":1}
I20260810 14:36:19.284113 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d41b8f6784405c9983deab6a8d08f2" candidate_uuid: "78b180891c294398a87c88625f4999de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c215b4e9937d4c75acc517f322cf3175" is_pre_election: true
I20260810 14:36:19.284113 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d41b8f6784405c9983deab6a8d08f2" candidate_uuid: "78b180891c294398a87c88625f4999de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b336ef0d9f4464ca4a69aad84c3c562" is_pre_election: true
I20260810 14:36:19.284699 5255 raft_consensus.cc:2454] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b180891c294398a87c88625f4999de in term 0.
I20260810 14:36:19.284699 5031 raft_consensus.cc:2454] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78b180891c294398a87c88625f4999de in term 0.
I20260810 14:36:19.285436 4688 leader_election.cc:302] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1b336ef0d9f4464ca4a69aad84c3c562, 78b180891c294398a87c88625f4999de, c215b4e9937d4c75acc517f322cf3175; no voters:
I20260810 14:36:19.285810 5376 raft_consensus.cc:2781] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:36:19.285954 5376 raft_consensus.cc:482] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:36:19.286082 5376 raft_consensus.cc:3037] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:19.288216 5376 raft_consensus.cc:504] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c215b4e9937d4c75acc517f322cf3175" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44751 } } peers { permanent_uuid: "1b336ef0d9f4464ca4a69aad84c3c562" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 40817 } } peers { permanent_uuid: "78b180891c294398a87c88625f4999de" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44311 } }
I20260810 14:36:19.288793 5376 leader_election.cc:288] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [CANDIDATE]: Term 1 election: Requested vote from peers c215b4e9937d4c75acc517f322cf3175 (127.6.46.195:44751), 1b336ef0d9f4464ca4a69aad84c3c562 (127.6.46.194:40817)
I20260810 14:36:19.289191 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d41b8f6784405c9983deab6a8d08f2" candidate_uuid: "78b180891c294398a87c88625f4999de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c215b4e9937d4c75acc517f322cf3175"
I20260810 14:36:19.289279 5031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3d41b8f6784405c9983deab6a8d08f2" candidate_uuid: "78b180891c294398a87c88625f4999de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b336ef0d9f4464ca4a69aad84c3c562"
I20260810 14:36:19.289356 5255 raft_consensus.cc:3037] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:19.289475 5031 raft_consensus.cc:3037] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:19.291482 5255 raft_consensus.cc:2454] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b180891c294398a87c88625f4999de in term 1.
I20260810 14:36:19.291755 5031 raft_consensus.cc:2454] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78b180891c294398a87c88625f4999de in term 1.
I20260810 14:36:19.291836 4690 leader_election.cc:302] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78b180891c294398a87c88625f4999de, c215b4e9937d4c75acc517f322cf3175; no voters:
I20260810 14:36:19.292104 5376 raft_consensus.cc:2781] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:19.292506 5376 raft_consensus.cc:686] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [term 1 LEADER]: Becoming Leader. State: Replica: 78b180891c294398a87c88625f4999de, State: Running, Role: LEADER
I20260810 14:36:19.292920 5376 consensus_queue.cc:237] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c215b4e9937d4c75acc517f322cf3175" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44751 } } peers { permanent_uuid: "1b336ef0d9f4464ca4a69aad84c3c562" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 40817 } } peers { permanent_uuid: "78b180891c294398a87c88625f4999de" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44311 } }
I20260810 14:36:19.297120 4547 catalog_manager.cc:5495] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de reported cstate change: term changed from 0 to 1, leader changed from <none> to 78b180891c294398a87c88625f4999de (127.6.46.193). New cstate: current_term: 1 leader_uuid: "78b180891c294398a87c88625f4999de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c215b4e9937d4c75acc517f322cf3175" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b336ef0d9f4464ca4a69aad84c3c562" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 40817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78b180891c294398a87c88625f4999de" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44311 } health_report { overall_health: HEALTHY } } }
I20260810 14:36:19.347774 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.342487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57848 (local address 127.6.46.193:44311)
0810 14:36:19.342688 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:19.342694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:19.342714 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:36:19.342911 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:19.342914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:19.342999 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:19.343152 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:36:19.343594 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.344364 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.345693 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.346134 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.346238 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.346730 (+ 492us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:36:19.347209 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:36:19.347340 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:19.347475 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 14:36:19.347519 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:36:19.355424 5031 raft_consensus.cc:1261] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562 [term 1 FOLLOWER]: Refusing update from remote peer 78b180891c294398a87c88625f4999de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:19.355422 5255 raft_consensus.cc:1261] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175 [term 1 FOLLOWER]: Refusing update from remote peer 78b180891c294398a87c88625f4999de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:19.356321 5382 consensus_queue.cc:1035] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b336ef0d9f4464ca4a69aad84c3c562" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 40817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:19.356621 5376 consensus_queue.cc:1035] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [LEADER]: Connected to new peer: Peer: permanent_uuid: "c215b4e9937d4c75acc517f322cf3175" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 44751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:19.366462 5386 mvcc.cc:205] Tried to move back new op lower bound from 7316982085030719488 to 7316982084787007488. Current Snapshot: MvccSnapshot[applied={T|T < 7316982085030719488}]
I20260810 14:36:19.380888 4546 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:19.381198 4546 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:19.385001 4546 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34b0b2552cb54dc196d4bb9e1f423a81: Sending DeleteTablet for 3 replicas of tablet c3d41b8f6784405c9983deab6a8d08f2
I20260810 14:36:19.386191 5222 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d41b8f6784405c9983deab6a8d08f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:19 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:44976
I20260810 14:36:19.386204 5002 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d41b8f6784405c9983deab6a8d08f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:19 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:33872
I20260810 14:36:19.386235 4769 tablet_service.cc:1514] Processing DeleteTablet for tablet c3d41b8f6784405c9983deab6a8d08f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:19 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:57840
I20260810 14:36:19.387955 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 4651
I20260810 14:36:19.394635 5404 tablet_replica.cc:330] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de: stopping tablet replica
I20260810 14:36:19.395073 5404 raft_consensus.cc:2229] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:36:19.395686 5404 raft_consensus.cc:2258] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:19.396433 5403 tablet_replica.cc:330] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562: stopping tablet replica
I20260810 14:36:19.396839 5403 raft_consensus.cc:2229] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:19.397374 5403 raft_consensus.cc:2258] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:19.401723 5404 ts_tablet_manager.cc:1905] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:19.410557 5404 ts_tablet_manager.cc:1918] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:19.411172 5403 ts_tablet_manager.cc:1905] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 14:36:19 dist-test-slave-q9h9 krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:36:19.416528 5404 log.cc:1196] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/c3d41b8f6784405c9983deab6a8d08f2
I20260810 14:36:19.417928 5403 ts_tablet_manager.cc:1918] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:19.421063 5403 log.cc:1196] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/c3d41b8f6784405c9983deab6a8d08f2
I20260810 14:36:19.421653 5403 ts_tablet_manager.cc:1939] T c3d41b8f6784405c9983deab6a8d08f2 P 1b336ef0d9f4464ca4a69aad84c3c562: Deleting consensus metadata
I20260810 14:36:19.423542 4529 catalog_manager.cc:4841] TS 1b336ef0d9f4464ca4a69aad84c3c562 (127.6.46.194:40817): tablet c3d41b8f6784405c9983deab6a8d08f2 (table test-table [id=fed44077db4b44c6b7c56d001373900b]) successfully deleted
I20260810 14:36:19.424088 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.389325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57862 (local address 127.6.46.193:44311)
0810 14:36:19.389493 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:19.389499 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:19.394856 (+ 5357us) server_negotiation.cc:407] Connection header received
0810 14:36:19.395109 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:19.395115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:19.395204 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:19.395265 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:19.397181 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.398245 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.401929 (+ 3684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.402602 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.402695 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.415573 (+ 12878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:19.415591 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:19.415609 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:19.415657 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:19.419567 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:19.420725 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:19.420734 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:19.420737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:19.420912 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:19.421479 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:19.421484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:19.421488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:19.422008 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:19.422334 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:19.423530 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0810 14:36:19.423752 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:36:19.424932 4861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:19.425714 5408 tablet_replica.cc:330] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175: stopping tablet replica
I20260810 14:36:19.426216 5408 raft_consensus.cc:2229] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:19.426781 5408 raft_consensus.cc:2258] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:19.429177 5408 ts_tablet_manager.cc:1905] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:19.436161 5408 ts_tablet_manager.cc:1918] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:19.436323 5408 log.cc:1196] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/c3d41b8f6784405c9983deab6a8d08f2
I20260810 14:36:19.436800 5408 ts_tablet_manager.cc:1939] T c3d41b8f6784405c9983deab6a8d08f2 P c215b4e9937d4c75acc517f322cf3175: Deleting consensus metadata
I20260810 14:36:19.438519 4530 catalog_manager.cc:4841] TS c215b4e9937d4c75acc517f322cf3175 (127.6.46.195:44751): tablet c3d41b8f6784405c9983deab6a8d08f2 (table test-table [id=fed44077db4b44c6b7c56d001373900b]) successfully deleted
I20260810 14:36:19.521905 5404 ts_tablet_manager.cc:1939] T c3d41b8f6784405c9983deab6a8d08f2 P 78b180891c294398a87c88625f4999de: Deleting consensus metadata
I20260810 14:36:19.522927 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 4651
W20260810 14:36:19.544442 4530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.46.193:44311 (error 108)
I20260810 14:36:19.545188 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 4884
I20260810 14:36:19.546376 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.545608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:44311 (local address 127.0.0.1:57868)
0810 14:36:19.546041 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:19.546170 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":194,"threads_started":1}
W20260810 14:36:19.546586 4530 catalog_manager.cc:4584] TS 78b180891c294398a87c88625f4999de (127.6.46.193:44311): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3d41b8f6784405c9983deab6a8d08f2: Network error: Client connection negotiation failed: client connection to 127.6.46.193:44311: connect: Connection refused (error 111)
Aug 10 14:36:19 dist-test-slave-q9h9 krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:36:19.559926 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.545930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.6.46.194:40817)
0810 14:36:19.546141 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:19.546145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:19.546238 (+ 93us) server_negotiation.cc:407] Connection header received
0810 14:36:19.546481 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:19.546487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:19.546575 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:19.546697 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:19.548014 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.548708 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.549990 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.550462 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.550580 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.554866 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:19.554878 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:19.554893 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:19.554924 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:19.557363 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:19.558041 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:19.558051 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:19.558055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:19.558210 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:19.558556 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:19.558561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:19.558565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:19.558953 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:19.559261 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:19.559505 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 14:36:19.559669 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260810 14:36:19.560498 5091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:19.600044 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.599625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:44311 (local address 127.0.0.1:57874)
0810 14:36:19.599817 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:19.599907 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 14:36:19.601096 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.600732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:44311 (local address 127.0.0.1:57882)
0810 14:36:19.600885 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:19.600956 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 14:36:19.648840 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 4884
I20260810 14:36:19.661574 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.661095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:44311 (local address 127.0.0.1:57886)
0810 14:36:19.661305 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:19.661406 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 14:36:19.662698 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.662319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:44311 (local address 127.0.0.1:57894)
0810 14:36:19.662451 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:19.662537 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 14:36:19.674299 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 5111
Aug 10 14:36:19 dist-test-slave-q9h9 krb5kdc[4477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:36:19.688918 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.675076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44988 (local address 127.6.46.195:44751)
0810 14:36:19.675235 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:19.675239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:19.675257 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:36:19.675415 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:19.675419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:19.675486 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:19.675566 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:19.676944 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.677645 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.679109 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:19.679497 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:19.679585 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:19.684154 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:19.684171 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:19.684187 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:19.684219 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:19.686761 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:19.687325 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:19.687332 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:19.687334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:19.687464 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:19.687741 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:19.687744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:19.687746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:19.688080 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:19.688311 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:19.688512 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 14:36:19.688679 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:36:19.689508 5323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:19.743371 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.742910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:44311 (local address 127.0.0.1:57904)
0810 14:36:19.743113 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:19.743217 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 14:36:19.744447 5413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:19.744022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:44311 (local address 127.0.0.1:57918)
0810 14:36:19.744243 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:19.744316 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:44311: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 14:36:19.776829 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 5111
I20260810 14:36:19.800941 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 4502
I20260810 14:36:19.802001 4594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:19.909976 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 4502
2026-08-10T14:36:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4505 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T14:36:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:19Z Disabled control of system clock
I20260810 14:36:19.997313 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43449
--webserver_interface=127.6.46.254
--webserver_port=0
--builtin_ntp_servers=127.6.46.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:43449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:36:20.315567 5425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:20.315815 5425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:20.315904 5425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:20.323130 5425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:20.323222 5425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:20.323266 5425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:20.323318 5425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:20.323361 5425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:20.331943 5425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:43449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:20.333000 5425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:20.334759 5425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:20.345563 5436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:20.345656 5437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:20.345623 5441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:20.346537 5425 server_base.cc:1034] running on GCE node
I20260810 14:36:20.347301 5425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:20.348526 5425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:20.349730 5425 hybrid_clock.cc:648] HybridClock initialized: now 1786372580349684 us; error 28 us; skew 500 ppm
I20260810 14:36:20.350066 5425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:36:20.352099 5425 webserver.cc:460] Webserver started at http://127.6.46.254:39727/ using document root <none> and password file <none>
I20260810 14:36:20.352717 5425 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:20.352818 5425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:20.353137 5425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:20.355904 5425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "c0feb58fb0e34733a9b44a2f0e5cdd3d"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-q9h9"
server_key: "7dec96a61cf6ac2c95d09a592299e567"
server_key_iv: "0372222521fea65644e0225ca34996d2"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.356660 5425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "c0feb58fb0e34733a9b44a2f0e5cdd3d"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-q9h9"
server_key: "7dec96a61cf6ac2c95d09a592299e567"
server_key_iv: "0372222521fea65644e0225ca34996d2"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.361465 5425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260810 14:36:20.364720 5451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.365870 5425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:36:20.366065 5425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "c0feb58fb0e34733a9b44a2f0e5cdd3d"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-q9h9"
server_key: "7dec96a61cf6ac2c95d09a592299e567"
server_key_iv: "0372222521fea65644e0225ca34996d2"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.366252 5425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:20.382437 5425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:20.383127 5425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:20.383376 5425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:20.399986 5425 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:43449
I20260810 14:36:20.400003 5530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:43449 every 8 connection(s)
I20260810 14:36:20.401173 5425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:20.404597 5532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:20.408115 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 5425
I20260810 14:36:20.408272 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:20.408687 6331 external_mini_cluster.cc:1428] Setting key 57c6bc8c36dc8606bffab07308b3cf4d
I20260810 14:36:20.416517 5532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d: Bootstrap starting.
I20260810 14:36:20.419883 5532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:20.420964 5532 log.cc:824] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:20.421301 5537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:20.410349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58540 (local address 127.6.46.254:43449)
0810 14:36:20.411225 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:20.411236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:20.411275 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:36:20.412794 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:20.412815 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:20.413247 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:20.413550 (+ 303us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:20.414144 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.415404 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.417166 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.417831 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.417974 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:20.419154 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:20.419164 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:20.419177 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:20.419661 (+ 484us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:20.419742 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:20.419763 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:20.419859 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:20.420120 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 14:36:20.420422 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":312,"threads_started":1}
I20260810 14:36:20.423527 5532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d: No bootstrap required, opened a new log
I20260810 14:36:20.430328 5532 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0feb58fb0e34733a9b44a2f0e5cdd3d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43449 } }
I20260810 14:36:20.430642 5532 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:20.430755 5532 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0feb58fb0e34733a9b44a2f0e5cdd3d, State: Initialized, Role: FOLLOWER
I20260810 14:36:20.431258 5532 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0feb58fb0e34733a9b44a2f0e5cdd3d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43449 } }
I20260810 14:36:20.431463 5532 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:20.431583 5532 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:20.431728 5532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:20.433710 5532 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0feb58fb0e34733a9b44a2f0e5cdd3d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43449 } }
I20260810 14:36:20.434219 5532 leader_election.cc:302] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0feb58fb0e34733a9b44a2f0e5cdd3d; no voters:
I20260810 14:36:20.434816 5532 leader_election.cc:288] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:20.434921 5539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:20.435868 5539 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [term 1 LEADER]: Becoming Leader. State: Replica: c0feb58fb0e34733a9b44a2f0e5cdd3d, State: Running, Role: LEADER
I20260810 14:36:20.436370 5539 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0feb58fb0e34733a9b44a2f0e5cdd3d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43449 } }
I20260810 14:36:20.436993 5532 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:20.438830 5543 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0feb58fb0e34733a9b44a2f0e5cdd3d. Latest consensus state: current_term: 1 leader_uuid: "c0feb58fb0e34733a9b44a2f0e5cdd3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0feb58fb0e34733a9b44a2f0e5cdd3d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43449 } } }
I20260810 14:36:20.439270 5543 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:20.441586 5551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:20.441452 5541 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0feb58fb0e34733a9b44a2f0e5cdd3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0feb58fb0e34733a9b44a2f0e5cdd3d" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43449 } } }
I20260810 14:36:20.441763 5541 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:20.446787 5551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:20.455771 5551 catalog_manager.cc:1347] Generated new cluster ID: ded57b69762644c2ba91d03bb2d67918
I20260810 14:36:20.455929 5551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:20.477788 5551 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:20.479533 5551 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:20.502110 5551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d: Generated new TSK 0
I20260810 14:36:20.502883 5551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:20.516275 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43449
--builtin_ntp_servers=127.6.46.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:36:20.841289 5571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:20.841532 5571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:20.841651 5571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:20.848832 5571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:20.848917 5571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:20.849053 5571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:36:20.857539 5571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:20.858644 5571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:20.860316 5571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:20.872401 5577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:20.872429 5578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:20.872941 5583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:20.873180 5571 server_base.cc:1034] running on GCE node
I20260810 14:36:20.873769 5571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:20.874670 5571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:20.875871 5571 hybrid_clock.cc:648] HybridClock initialized: now 1786372580875832 us; error 28 us; skew 500 ppm
I20260810 14:36:20.876159 5571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:36:20.878252 5571 webserver.cc:460] Webserver started at http://127.6.46.193:35831/ using document root <none> and password file <none>
I20260810 14:36:20.878866 5571 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:20.878961 5571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:20.879252 5571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:20.881829 5571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "2031dce0fd7849d4b7a2ce5f3df73368"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-q9h9"
server_key: "e0cd69ff611b81d6b0901ca6023a3dc4"
server_key_iv: "3c580a099749961dedbbb27d9c2b1dcc"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.882630 5571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "2031dce0fd7849d4b7a2ce5f3df73368"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-q9h9"
server_key: "e0cd69ff611b81d6b0901ca6023a3dc4"
server_key_iv: "3c580a099749961dedbbb27d9c2b1dcc"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.887228 5571 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 14:36:20.890463 5592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.891610 5571 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260810 14:36:20.891750 5571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "2031dce0fd7849d4b7a2ce5f3df73368"
format_stamp: "Formatted at 2026-08-10 14:36:20 on dist-test-slave-q9h9"
server_key: "e0cd69ff611b81d6b0901ca6023a3dc4"
server_key_iv: "3c580a099749961dedbbb27d9c2b1dcc"
server_key_version: "encryptionkey@0"
I20260810 14:36:20.891938 5571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:20.905401 5571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:20.906113 5571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:20.906376 5571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:20.907230 5571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:20.908532 5571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:20.908619 5571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.908722 5571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:20.908797 5571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:20.937739 5571 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:39765
I20260810 14:36:20.937808 5774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:39765 every 8 connection(s)
I20260810 14:36:20.939018 5571 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:36:20.945446 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 5571
I20260810 14:36:20.945601 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:36:20.946008 6331 external_mini_cluster.cc:1428] Setting key cae743d54b31abfc9aba368c281017ee
I20260810 14:36:20.948532 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43449
--builtin_ntp_servers=127.6.46.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:36:20.951122 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:20.941397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:52173 (local address 127.6.46.254:43449)
0810 14:36:20.942340 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:20.942344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:20.943021 (+ 677us) server_negotiation.cc:407] Connection header received
0810 14:36:20.944844 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:20.944848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:20.944913 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:20.945014 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:20.946094 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.946888 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.948452 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.948937 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.949049 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:20.950331 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:20.950336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:20.950339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:20.950357 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:20.950399 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:20.950411 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:20.950485 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:20.950703 (+ 218us) server_negotiation.cc:299] Negotiation successful
0810 14:36:20.950797 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":753,"threads_started":1}
I20260810 14:36:20.951977 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:20.941867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:43449 (local address 127.6.46.193:52173)
0810 14:36:20.942868 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:20.942920 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:20.944598 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:20.945215 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:20.945222 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:20.945326 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:20.945911 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:20.945929 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.947025 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.947031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:20.948311 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:20.948324 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:20.949150 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:20.949168 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:20.949377 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:20.949995 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:20.949998 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:20.950072 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:20.950142 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:20.950497 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:20.950509 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:20.950517 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 14:36:20.950661 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 14:36:20.951008 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":339,"threads_started":1}
I20260810 14:36:20.953343 5778 heartbeater.cc:347] Connected to a master server at 127.6.46.254:43449
I20260810 14:36:20.953601 5778 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:20.954322 5778 heartbeater.cc:511] Master 127.6.46.254:43449 requested a full tablet report, sending...
I20260810 14:36:20.956387 5481 ts_manager.cc:194] Registered new tserver with Master: 2031dce0fd7849d4b7a2ce5f3df73368 (127.6.46.193:39765)
I20260810 14:36:20.958197 5481 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.46.193:52173
W20260810 14:36:21.270642 5785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:21.270905 5785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:21.271035 5785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:21.278182 5785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:21.278275 5785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:21.278415 5785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:36:21.287138 5785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:21.288282 5785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:21.289947 5785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:21.302282 5791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:21.302338 5795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:21.302317 5792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:21.303323 5785 server_base.cc:1034] running on GCE node
I20260810 14:36:21.303898 5785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:21.304775 5785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:21.305976 5785 hybrid_clock.cc:648] HybridClock initialized: now 1786372581305930 us; error 33 us; skew 500 ppm
I20260810 14:36:21.306296 5785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:36:21.308389 5785 webserver.cc:460] Webserver started at http://127.6.46.194:32959/ using document root <none> and password file <none>
I20260810 14:36:21.309010 5785 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:21.309111 5785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:21.309422 5785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:21.312140 5785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "89f55b53bf2543e2a4c359aa50304ec5"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-q9h9"
server_key: "5e04f47003fd0240d9f0c6379fef0839"
server_key_iv: "106aa1f6f966f7581b145ca27939315a"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.312886 5785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "89f55b53bf2543e2a4c359aa50304ec5"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-q9h9"
server_key: "5e04f47003fd0240d9f0c6379fef0839"
server_key_iv: "106aa1f6f966f7581b145ca27939315a"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.317695 5785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 14:36:21.320922 5803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.322124 5785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:36:21.322280 5785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "89f55b53bf2543e2a4c359aa50304ec5"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-q9h9"
server_key: "5e04f47003fd0240d9f0c6379fef0839"
server_key_iv: "106aa1f6f966f7581b145ca27939315a"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.322469 5785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:21.340045 5785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:21.340726 5785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:21.340998 5785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:21.341866 5785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:21.343478 5785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:21.343578 5785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.343679 5785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:21.343750 5785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.373198 5785 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:38451
I20260810 14:36:21.373243 5987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:38451 every 8 connection(s)
I20260810 14:36:21.374456 5785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:36:21.377187 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 5785
I20260810 14:36:21.377324 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:36:21.377739 6331 external_mini_cluster.cc:1428] Setting key 742ede5a29d7286af3daec1db5c52213
I20260810 14:36:21.380611 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43449
--builtin_ntp_servers=127.6.46.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:36:21.386657 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.376809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:56859 (local address 127.6.46.254:43449)
0810 14:36:21.377032 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:21.377038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:21.378555 (+ 1517us) server_negotiation.cc:407] Connection header received
0810 14:36:21.380487 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:21.380493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:21.380572 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:21.380685 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:21.381703 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.382712 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.384177 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.384676 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.384790 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.385930 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:21.385936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:21.385941 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:21.385961 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:21.385997 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:21.386011 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:21.386114 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:21.386277 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 14:36:21.386373 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 14:36:21.387717 5992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.377333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:43449 (local address 127.6.46.194:56859)
0810 14:36:21.378392 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:21.378435 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:21.380233 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:21.380910 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:21.380921 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:21.381004 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:21.381541 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.381556 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.382837 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.382841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.384030 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.384042 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.384846 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.384850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.385036 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.385640 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:21.385643 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:21.385687 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:21.385762 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:21.386107 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:21.386114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:21.386120 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 14:36:21.386250 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 14:36:21.386518 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":344,"threads_started":1}
I20260810 14:36:21.388934 5988 heartbeater.cc:347] Connected to a master server at 127.6.46.254:43449
I20260810 14:36:21.389230 5988 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:21.390074 5988 heartbeater.cc:511] Master 127.6.46.254:43449 requested a full tablet report, sending...
I20260810 14:36:21.391458 5481 ts_manager.cc:194] Registered new tserver with Master: 89f55b53bf2543e2a4c359aa50304ec5 (127.6.46.194:38451)
I20260810 14:36:21.392227 5481 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.46.194:56859
W20260810 14:36:21.704125 5995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:21.704372 5995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:21.704474 5995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:21.711410 5995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:21.711495 5995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:21.711627 5995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:36:21.720395 5995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:21.721662 5995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:21.723330 5995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:21.735553 6008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:21.735656 6010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:21.736289 6012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:21.737818 5995 server_base.cc:1034] running on GCE node
I20260810 14:36:21.738466 5995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:21.739332 5995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:21.740514 5995 hybrid_clock.cc:648] HybridClock initialized: now 1786372581740456 us; error 54 us; skew 500 ppm
I20260810 14:36:21.740800 5995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:36:21.743045 5995 webserver.cc:460] Webserver started at http://127.6.46.195:33405/ using document root <none> and password file <none>
I20260810 14:36:21.743691 5995 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:21.743785 5995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:21.744105 5995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:21.746986 5995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "287ed7cf08c14852a8c7fd8ad758e172"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-q9h9"
server_key: "3d5d7ffd0ebe009e7c2b66296539e289"
server_key_iv: "1a6974340676fe14b1b8f409c21b77f5"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.747723 5995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "287ed7cf08c14852a8c7fd8ad758e172"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-q9h9"
server_key: "3d5d7ffd0ebe009e7c2b66296539e289"
server_key_iv: "1a6974340676fe14b1b8f409c21b77f5"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.752401 5995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:36:21.755693 6020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.756856 5995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:36:21.757000 5995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "287ed7cf08c14852a8c7fd8ad758e172"
format_stamp: "Formatted at 2026-08-10 14:36:21 on dist-test-slave-q9h9"
server_key: "3d5d7ffd0ebe009e7c2b66296539e289"
server_key_iv: "1a6974340676fe14b1b8f409c21b77f5"
server_key_version: "encryptionkey@0"
I20260810 14:36:21.757186 5995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:21.773511 5995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:21.774236 5995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:21.774503 5995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:21.775365 5995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:21.776710 5995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:21.776810 5995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.776916 5995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:21.776986 5995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:21.806525 5995 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:38987
I20260810 14:36:21.806582 6215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:38987 every 8 connection(s)
I20260810 14:36:21.807778 5995 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:36:21.809214 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 5995
I20260810 14:36:21.809369 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:36:21.809877 6331 external_mini_cluster.cc:1428] Setting key 177755d724942ab456014c034f13c8a3
I20260810 14:36:21.819168 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.810167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:49505 (local address 127.6.46.254:43449)
0810 14:36:21.810392 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:21.810397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:21.812014 (+ 1617us) server_negotiation.cc:407] Connection header received
0810 14:36:21.813696 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:21.813701 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:21.813766 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:21.813845 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:21.814827 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.815544 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.816984 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.817364 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.817450 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.818571 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:21.818576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:21.818579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:21.818596 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:21.818625 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:21.818637 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:21.818711 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:21.818870 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 14:36:21.818944 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 14:36:21.819988 6222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.810738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:43449 (local address 127.6.46.195:49505)
0810 14:36:21.811863 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:21.811907 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:21.813459 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:21.814040 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:21.814052 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:21.814149 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:21.814682 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.814698 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.815666 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.815669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.816880 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.816892 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.817518 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.817522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.817710 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.818311 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:21.818313 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:21.818350 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:21.818422 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:21.818715 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:21.818721 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:21.818725 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 14:36:21.818843 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 14:36:21.819111 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":461,"threads_started":1}
I20260810 14:36:21.821205 6216 heartbeater.cc:347] Connected to a master server at 127.6.46.254:43449
I20260810 14:36:21.821453 6216 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:21.822150 6216 heartbeater.cc:511] Master 127.6.46.254:43449 requested a full tablet report, sending...
I20260810 14:36:21.823473 5481 ts_manager.cc:194] Registered new tserver with Master: 287ed7cf08c14852a8c7fd8ad758e172 (127.6.46.195:38987)
I20260810 14:36:21.824447 5481 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.46.195:49505
I20260810 14:36:21.824754 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:36:21.834475 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.829359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58556 (local address 127.6.46.254:43449)
0810 14:36:21.829494 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:21.829498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:21.829925 (+ 427us) server_negotiation.cc:407] Connection header received
0810 14:36:21.830059 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:21.830072 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:21.830139 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:21.830211 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:21.830644 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.831288 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.832552 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.832922 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.833005 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.833926 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:21.833931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:21.833934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:21.833949 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:21.833974 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:21.833984 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:21.834123 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:21.834233 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 14:36:21.834273 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 14:36:21.837805 5481 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:36:21.844399 5481 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:36:21.866720 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.855639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:38987 (local address 127.0.0.1:39670)
0810 14:36:21.857387 (+ 1748us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:21.857405 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:21.857523 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:21.858747 (+ 1224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:21.858754 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:21.858789 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:21.859263 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.859275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.861301 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.861307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.863677 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.863692 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.865390 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.865397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.865574 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.866169 (+ 595us) client_negotiation.cc:769] Sending connection context
0810 14:36:21.866244 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 14:36:21.866379 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1610,"mutex_wait_us":3,"thread_start_us":286,"threads_started":1}
I20260810 14:36:21.867046 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.857786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.6.46.194:38451)
0810 14:36:21.858359 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:21.858376 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:21.859124 (+ 748us) server_negotiation.cc:407] Connection header received
0810 14:36:21.859417 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:21.859424 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:21.859633 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:21.859791 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:21.860653 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.862403 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.864590 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.865587 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.865738 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.866001 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:21.866492 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 14:36:21.866673 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":275,"threads_started":1}
I20260810 14:36:21.867430 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.855949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.6.46.195:38987)
0810 14:36:21.858087 (+ 2138us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:21.858096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:21.858129 (+ 33us) server_negotiation.cc:407] Connection header received
0810 14:36:21.858247 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:21.858255 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:21.858492 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:21.859437 (+ 945us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:21.859709 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.861163 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.863864 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.865197 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.865319 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.865601 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:21.866547 (+ 946us) server_negotiation.cc:299] Negotiation successful
0810 14:36:21.866724 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1960,"thread_start_us":1806,"threads_started":1}
I20260810 14:36:21.867579 6245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.857138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:38451 (local address 127.0.0.1:50714)
0810 14:36:21.859001 (+ 1863us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:21.859017 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:21.859230 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:21.860122 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:21.860127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:21.860177 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:21.860480 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.860490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.862574 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.862579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.864423 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.864438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.865783 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.865790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.865982 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.866040 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 14:36:21.866152 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 14:36:21.866330 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1736,"mutex_wait_us":26,"thread_start_us":1531,"threads_started":1}
I20260810 14:36:21.868631 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.855413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39765 (local address 127.0.0.1:55406)
0810 14:36:21.856351 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:21.856403 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:21.856566 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:21.861478 (+ 4912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:21.861486 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:21.861509 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:21.861964 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.861979 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.863512 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.863517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.865848 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.865865 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.867919 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.867925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.868098 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.868128 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 14:36:21.868216 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 14:36:21.868328 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":1,"thread_start_us":1695,"threads_started":1}
I20260810 14:36:21.868829 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.855632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55406 (local address 127.6.46.193:39765)
0810 14:36:21.860978 (+ 5346us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:21.860986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:21.861010 (+ 24us) server_negotiation.cc:407] Connection header received
0810 14:36:21.861109 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:21.861115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:21.861316 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:21.861475 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:21.862180 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.863377 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.866104 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.867465 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.867589 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.867799 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:21.868298 (+ 499us) server_negotiation.cc:299] Negotiation successful
0810 14:36:21.868460 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5200,"thread_start_us":5079,"threads_started":1}
I20260810 14:36:21.871328 5882 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb3cbf8ffb44dabbda95a2b0733cac3 (DEFAULT_TABLE table=test-table [id=3d47bd488e994fefae795faebc007649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:21.871099 6114 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb3cbf8ffb44dabbda95a2b0733cac3 (DEFAULT_TABLE table=test-table [id=3d47bd488e994fefae795faebc007649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:21.872467 5882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb3cbf8ffb44dabbda95a2b0733cac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:21.872584 6114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb3cbf8ffb44dabbda95a2b0733cac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:21.872323 5671 tablet_service.cc:1467] Processing CreateTablet for tablet 9bb3cbf8ffb44dabbda95a2b0733cac3 (DEFAULT_TABLE table=test-table [id=3d47bd488e994fefae795faebc007649]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:21.873556 5671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bb3cbf8ffb44dabbda95a2b0733cac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:21.883415 6251 tablet_bootstrap.cc:492] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5: Bootstrap starting.
I20260810 14:36:21.886509 6251 tablet_bootstrap.cc:654] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:21.886605 6254 tablet_bootstrap.cc:492] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172: Bootstrap starting.
I20260810 14:36:21.888159 6251 log.cc:824] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:21.889333 6257 tablet_bootstrap.cc:492] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368: Bootstrap starting.
I20260810 14:36:21.890162 6254 tablet_bootstrap.cc:654] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:21.891386 6254 log.cc:824] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:21.892095 6257 tablet_bootstrap.cc:654] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:21.893191 6257 log.cc:824] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:21.894356 6254 tablet_bootstrap.cc:492] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172: No bootstrap required, opened a new log
I20260810 14:36:21.894376 6251 tablet_bootstrap.cc:492] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5: No bootstrap required, opened a new log
I20260810 14:36:21.894623 6251 ts_tablet_manager.cc:1397] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260810 14:36:21.894623 6254 ts_tablet_manager.cc:1397] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260810 14:36:21.896437 6257 tablet_bootstrap.cc:492] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368: No bootstrap required, opened a new log
I20260810 14:36:21.896675 6257 ts_tablet_manager.cc:1397] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 14:36:21.903259 6257 raft_consensus.cc:348] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2031dce0fd7849d4b7a2ce5f3df73368" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39765 } } peers { permanent_uuid: "287ed7cf08c14852a8c7fd8ad758e172" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 38987 } } peers { permanent_uuid: "89f55b53bf2543e2a4c359aa50304ec5" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38451 } }
I20260810 14:36:21.904196 6257 raft_consensus.cc:374] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:21.904363 6257 raft_consensus.cc:729] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2031dce0fd7849d4b7a2ce5f3df73368, State: Initialized, Role: FOLLOWER
I20260810 14:36:21.904633 6251 raft_consensus.cc:348] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2031dce0fd7849d4b7a2ce5f3df73368" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39765 } } peers { permanent_uuid: "287ed7cf08c14852a8c7fd8ad758e172" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 38987 } } peers { permanent_uuid: "89f55b53bf2543e2a4c359aa50304ec5" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38451 } }
I20260810 14:36:21.904633 6254 raft_consensus.cc:348] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2031dce0fd7849d4b7a2ce5f3df73368" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39765 } } peers { permanent_uuid: "287ed7cf08c14852a8c7fd8ad758e172" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 38987 } } peers { permanent_uuid: "89f55b53bf2543e2a4c359aa50304ec5" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38451 } }
I20260810 14:36:21.905035 6251 raft_consensus.cc:374] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:21.905036 6254 raft_consensus.cc:374] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:21.905138 6251 raft_consensus.cc:729] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89f55b53bf2543e2a4c359aa50304ec5, State: Initialized, Role: FOLLOWER
I20260810 14:36:21.905138 6254 raft_consensus.cc:729] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287ed7cf08c14852a8c7fd8ad758e172, State: Initialized, Role: FOLLOWER
I20260810 14:36:21.904974 6257 consensus_queue.cc:260] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2031dce0fd7849d4b7a2ce5f3df73368" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39765 } } peers { permanent_uuid: "287ed7cf08c14852a8c7fd8ad758e172" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 38987 } } peers { permanent_uuid: "89f55b53bf2543e2a4c359aa50304ec5" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38451 } }
I20260810 14:36:21.905683 6254 consensus_queue.cc:260] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2031dce0fd7849d4b7a2ce5f3df73368" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39765 } } peers { permanent_uuid: "287ed7cf08c14852a8c7fd8ad758e172" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 38987 } } peers { permanent_uuid: "89f55b53bf2543e2a4c359aa50304ec5" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38451 } }
I20260810 14:36:21.905682 6251 consensus_queue.cc:260] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2031dce0fd7849d4b7a2ce5f3df73368" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39765 } } peers { permanent_uuid: "287ed7cf08c14852a8c7fd8ad758e172" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 38987 } } peers { permanent_uuid: "89f55b53bf2543e2a4c359aa50304ec5" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38451 } }
I20260810 14:36:21.907251 5778 heartbeater.cc:503] Master 127.6.46.254:43449 was elected leader, sending a full tablet report...
I20260810 14:36:21.907914 6257 ts_tablet_manager.cc:1428] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260810 14:36:21.908850 5988 heartbeater.cc:503] Master 127.6.46.254:43449 was elected leader, sending a full tablet report...
I20260810 14:36:21.909363 6216 heartbeater.cc:503] Master 127.6.46.254:43449 was elected leader, sending a full tablet report...
I20260810 14:36:21.910718 6251 ts_tablet_manager.cc:1428] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260810 14:36:21.911019 6254 ts_tablet_manager.cc:1428] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260810 14:36:21.940995 5779 tablet.cc:2369] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:36:21.988149 6264 raft_consensus.cc:482] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:36:21.988411 6264 raft_consensus.cc:504] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2031dce0fd7849d4b7a2ce5f3df73368" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39765 } } peers { permanent_uuid: "287ed7cf08c14852a8c7fd8ad758e172" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 38987 } } peers { permanent_uuid: "89f55b53bf2543e2a4c359aa50304ec5" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38451 } }
I20260810 14:36:21.989981 6264 leader_election.cc:288] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2031dce0fd7849d4b7a2ce5f3df73368 (127.6.46.193:39765), 287ed7cf08c14852a8c7fd8ad758e172 (127.6.46.195:38987)
I20260810 14:36:21.998538 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.990418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:49943 (local address 127.6.46.193:39765)
0810 14:36:21.990601 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:21.990607 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:21.991320 (+ 713us) server_negotiation.cc:407] Connection header received
0810 14:36:21.991455 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:21.991460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:21.991548 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:21.991632 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:21.992373 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.993437 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.996375 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.997556 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.997705 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.997771 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:21.998180 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 14:36:21.998299 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:36:21.998538 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.990471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:34641 (local address 127.6.46.195:38987)
0810 14:36:21.990655 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:21.990660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:21.991246 (+ 586us) server_negotiation.cc:407] Connection header received
0810 14:36:21.991454 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:21.991460 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:21.991536 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:21.991632 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:21.992373 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.993438 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.996383 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.997571 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.997725 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.997783 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:21.998174 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 14:36:21.998300 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 14:36:21.998740 6269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.990234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:38987 (local address 127.6.46.194:34641)
0810 14:36:21.991135 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:21.991155 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:21.991307 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:21.991654 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:21.991659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:21.991745 (+ 86us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:21.992251 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.992261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.993584 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.993588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.996166 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.996183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.997725 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.997731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.997978 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.998056 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 14:36:21.998148 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 14:36:21.998373 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":224,"spinlock_wait_cycles":19328,"thread_start_us":534,"threads_started":1}
I20260810 14:36:21.998842 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:21.990306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39765 (local address 127.6.46.194:49943)
0810 14:36:21.991223 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:21.991237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:21.991331 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:21.991654 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:21.991659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:21.991686 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:21.992203 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.992220 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.993582 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.993588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.996169 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:21.996183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:21.997766 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:21.997772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:21.997976 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:21.998055 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 14:36:21.998148 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 14:36:21.998373 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":253,"thread_start_us":538,"threads_started":1}
I20260810 14:36:21.999778 5701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb3cbf8ffb44dabbda95a2b0733cac3" candidate_uuid: "89f55b53bf2543e2a4c359aa50304ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2031dce0fd7849d4b7a2ce5f3df73368" is_pre_election: true
I20260810 14:36:22.000277 5701 raft_consensus.cc:2454] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f55b53bf2543e2a4c359aa50304ec5 in term 0.
I20260810 14:36:22.000584 6145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb3cbf8ffb44dabbda95a2b0733cac3" candidate_uuid: "89f55b53bf2543e2a4c359aa50304ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287ed7cf08c14852a8c7fd8ad758e172" is_pre_election: true
I20260810 14:36:22.000954 5805 leader_election.cc:302] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2031dce0fd7849d4b7a2ce5f3df73368, 89f55b53bf2543e2a4c359aa50304ec5; no voters:
I20260810 14:36:22.001135 6145 raft_consensus.cc:2454] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89f55b53bf2543e2a4c359aa50304ec5 in term 0.
I20260810 14:36:22.001355 6264 raft_consensus.cc:2781] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:36:22.001489 6264 raft_consensus.cc:482] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:36:22.001597 6264 raft_consensus.cc:3037] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:22.003800 6264 raft_consensus.cc:504] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2031dce0fd7849d4b7a2ce5f3df73368" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39765 } } peers { permanent_uuid: "287ed7cf08c14852a8c7fd8ad758e172" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 38987 } } peers { permanent_uuid: "89f55b53bf2543e2a4c359aa50304ec5" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38451 } }
I20260810 14:36:22.004323 6264 leader_election.cc:288] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [CANDIDATE]: Term 1 election: Requested vote from peers 2031dce0fd7849d4b7a2ce5f3df73368 (127.6.46.193:39765), 287ed7cf08c14852a8c7fd8ad758e172 (127.6.46.195:38987)
I20260810 14:36:22.004681 5701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb3cbf8ffb44dabbda95a2b0733cac3" candidate_uuid: "89f55b53bf2543e2a4c359aa50304ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2031dce0fd7849d4b7a2ce5f3df73368"
I20260810 14:36:22.004760 6145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bb3cbf8ffb44dabbda95a2b0733cac3" candidate_uuid: "89f55b53bf2543e2a4c359aa50304ec5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "287ed7cf08c14852a8c7fd8ad758e172"
I20260810 14:36:22.004846 5701 raft_consensus.cc:3037] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:22.004930 6145 raft_consensus.cc:3037] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:22.007014 5701 raft_consensus.cc:2454] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f55b53bf2543e2a4c359aa50304ec5 in term 1.
I20260810 14:36:22.007144 6145 raft_consensus.cc:2454] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89f55b53bf2543e2a4c359aa50304ec5 in term 1.
I20260810 14:36:22.007398 5805 leader_election.cc:302] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2031dce0fd7849d4b7a2ce5f3df73368, 89f55b53bf2543e2a4c359aa50304ec5; no voters:
I20260810 14:36:22.007675 6264 raft_consensus.cc:2781] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:22.008074 6264 raft_consensus.cc:686] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [term 1 LEADER]: Becoming Leader. State: Replica: 89f55b53bf2543e2a4c359aa50304ec5, State: Running, Role: LEADER
I20260810 14:36:22.008498 6264 consensus_queue.cc:237] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2031dce0fd7849d4b7a2ce5f3df73368" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39765 } } peers { permanent_uuid: "287ed7cf08c14852a8c7fd8ad758e172" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 38987 } } peers { permanent_uuid: "89f55b53bf2543e2a4c359aa50304ec5" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38451 } }
I20260810 14:36:22.012614 5481 catalog_manager.cc:5495] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 89f55b53bf2543e2a4c359aa50304ec5 (127.6.46.194). New cstate: current_term: 1 leader_uuid: "89f55b53bf2543e2a4c359aa50304ec5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2031dce0fd7849d4b7a2ce5f3df73368" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287ed7cf08c14852a8c7fd8ad758e172" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 38987 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89f55b53bf2543e2a4c359aa50304ec5" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 38451 } health_report { overall_health: HEALTHY } } }
W20260810 14:36:22.059427 6220 tablet.cc:2369] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:36:22.064554 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:22.059311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.6.46.194:38451)
0810 14:36:22.059470 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:22.059474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:22.059493 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:36:22.059583 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:22.059587 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:22.059703 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:22.059824 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:36:22.060338 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:22.061055 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:22.062511 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:22.062919 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:22.063011 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:22.063506 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:36:22.063972 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:36:22.064098 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:22.064270 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 14:36:22.064313 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 14:36:22.073087 6145 raft_consensus.cc:1261] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172 [term 1 FOLLOWER]: Refusing update from remote peer 89f55b53bf2543e2a4c359aa50304ec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:22.073067 5701 raft_consensus.cc:1261] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368 [term 1 FOLLOWER]: Refusing update from remote peer 89f55b53bf2543e2a4c359aa50304ec5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:22.074142 6264 consensus_queue.cc:1035] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2031dce0fd7849d4b7a2ce5f3df73368" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 39765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:22.074425 6274 consensus_queue.cc:1035] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "287ed7cf08c14852a8c7fd8ad758e172" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 38987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:22.087544 6282 mvcc.cc:205] Tried to move back new op lower bound from 7316982096160272384 to 7316982095910359040. Current Snapshot: MvccSnapshot[applied={T|T < 7316982096160272384}]
I20260810 14:36:22.098271 5481 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:22.098532 5481 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:22.102111 5481 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0feb58fb0e34733a9b44a2f0e5cdd3d: Sending DeleteTablet for 3 replicas of tablet 9bb3cbf8ffb44dabbda95a2b0733cac3
I20260810 14:36:22.103281 6114 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb3cbf8ffb44dabbda95a2b0733cac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:22 UTC) from {username='slave'} at 127.0.0.1:39670
I20260810 14:36:22.103412 5671 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb3cbf8ffb44dabbda95a2b0733cac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:22 UTC) from {username='slave'} at 127.0.0.1:55406
I20260810 14:36:22.104074 5882 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bb3cbf8ffb44dabbda95a2b0733cac3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:22 UTC) from {username='slave'} at 127.0.0.1:50714
I20260810 14:36:22.104149 6296 tablet_replica.cc:330] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172: stopping tablet replica
I20260810 14:36:22.104530 6296 raft_consensus.cc:2229] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:22.104674 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 5571
I20260810 14:36:22.105057 6296 raft_consensus.cc:2258] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:22.107275 6296 ts_tablet_manager.cc:1905] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:22.114507 6296 ts_tablet_manager.cc:1918] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:22.114727 6296 log.cc:1196] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/9bb3cbf8ffb44dabbda95a2b0733cac3
I20260810 14:36:22.115319 6296 ts_tablet_manager.cc:1939] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 287ed7cf08c14852a8c7fd8ad758e172: Deleting consensus metadata
I20260810 14:36:22.117275 5457 catalog_manager.cc:4841] TS 287ed7cf08c14852a8c7fd8ad758e172 (127.6.46.195:38987): tablet 9bb3cbf8ffb44dabbda95a2b0733cac3 (table test-table [id=3d47bd488e994fefae795faebc007649]) successfully deleted
I20260810 14:36:22.118316 6300 tablet_replica.cc:330] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5: stopping tablet replica
I20260810 14:36:22.118419 6299 tablet_replica.cc:330] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368: stopping tablet replica
I20260810 14:36:22.118754 6300 raft_consensus.cc:2229] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:36:22.119498 6300 raft_consensus.cc:2258] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:22.118826 6299 raft_consensus.cc:2229] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:22.120790 6299 raft_consensus.cc:2258] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:22.121551 6300 ts_tablet_manager.cc:1905] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:22.123011 6299 ts_tablet_manager.cc:1905] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 2031dce0fd7849d4b7a2ce5f3df73368: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:22.124346 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:22.107996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55410 (local address 127.6.46.193:39765)
0810 14:36:22.108178 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:22.108184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:22.116278 (+ 8094us) server_negotiation.cc:407] Connection header received
0810 14:36:22.116736 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:22.116741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:22.116841 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:22.116918 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:22.117716 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:22.118853 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:22.120287 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:22.120933 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:22.121060 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:22.122469 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:22.122480 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:22.122495 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:22.123196 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:22.123261 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:22.123284 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:22.123389 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:22.123863 (+ 474us) server_negotiation.cc:299] Negotiation successful
0810 14:36:22.123969 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 14:36:22.125118 5763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:22.129916 6300 ts_tablet_manager.cc:1918] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:22.130123 6300 log.cc:1196] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/9bb3cbf8ffb44dabbda95a2b0733cac3
I20260810 14:36:22.130688 6300 ts_tablet_manager.cc:1939] T 9bb3cbf8ffb44dabbda95a2b0733cac3 P 89f55b53bf2543e2a4c359aa50304ec5: Deleting consensus metadata
I20260810 14:36:22.132407 5457 catalog_manager.cc:4841] TS 89f55b53bf2543e2a4c359aa50304ec5 (127.6.46.194:38451): tablet 9bb3cbf8ffb44dabbda95a2b0733cac3 (table test-table [id=3d47bd488e994fefae795faebc007649]) successfully deleted
I20260810 14:36:22.212864 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 5571
I20260810 14:36:22.242298 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 5785
W20260810 14:36:22.242360 5454 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.46.193:39765 (error 108)
I20260810 14:36:22.244692 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:22.244201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39765 (local address 127.0.0.1:55412)
0810 14:36:22.244394 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:22.244522 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 14:36:22.245106 5454 catalog_manager.cc:4584] TS 2031dce0fd7849d4b7a2ce5f3df73368 (127.6.46.193:39765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9bb3cbf8ffb44dabbda95a2b0733cac3: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39765: connect: Connection refused (error 111)
I20260810 14:36:22.251523 6246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:22.243071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.6.46.194:38451)
0810 14:36:22.243686 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:22.243692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:22.243712 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:36:22.243783 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:22.243789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:22.243892 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:22.243997 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:22.244695 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:22.246540 (+ 1845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:22.248470 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:22.248897 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:22.248994 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:22.250149 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:22.250158 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:22.250171 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:22.250790 (+ 619us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:22.250835 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:22.250855 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:22.250938 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:22.251092 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 14:36:22.251177 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478}
I20260810 14:36:22.252251 5982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:22.299003 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:22.298494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39765 (local address 127.0.0.1:55416)
0810 14:36:22.298728 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:22.298825 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260810 14:36:22.300055 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:22.299678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39765 (local address 127.0.0.1:55432)
0810 14:36:22.299842 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:22.299917 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 14:36:22.360422 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 5785
I20260810 14:36:22.360486 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:22.360018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39765 (local address 127.0.0.1:55438)
0810 14:36:22.360215 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:22.360296 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 14:36:22.362171 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:22.361734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39765 (local address 127.0.0.1:55442)
0810 14:36:22.361890 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:22.361982 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 14:36:22.383141 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 5995
I20260810 14:36:22.393658 6242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:22.383864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39678 (local address 127.6.46.195:38987)
0810 14:36:22.384019 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:22.384023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:22.384079 (+ 56us) server_negotiation.cc:407] Connection header received
0810 14:36:22.384222 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:22.384226 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:22.384307 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:22.384390 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:22.384706 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:22.385421 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:22.386687 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:22.387066 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:22.387165 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:22.392526 (+ 5361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:22.392535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:22.392549 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:22.393088 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:22.393157 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:22.393178 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:22.393262 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:22.393343 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 14:36:22.393429 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:36:22.394276 6210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:22.442694 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:22.442273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39765 (local address 127.0.0.1:55456)
0810 14:36:22.442458 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:22.442544 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 14:36:22.443748 6236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:22.443349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:39765 (local address 127.0.0.1:55458)
0810 14:36:22.443527 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:22.443615 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.193:39765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 14:36:22.479809 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 5995
I20260810 14:36:22.501571 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 5425
I20260810 14:36:22.502648 5525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:22.607239 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 5425
2026-08-10T14:36:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2696 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:22 dist-test-slave-q9h9 krb5kdc[6310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:22 dist-test-slave-q9h9 krb5kdc[6310](info): set up 2 sockets
Aug 10 14:36:22 dist-test-slave-q9h9 krb5kdc[6310](info): commencing operation
krb5kdc: starting...
W20260810 14:36:23.835240 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.180s user 0.000s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:24 dist-test-slave-q9h9 krb5kdc[6310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:36:24.095418 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:37319
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:37319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:24.416904 6342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:24.417197 6342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:24.417333 6342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:24.424942 6342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:24.425050 6342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:24.425125 6342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:24.425209 6342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:24.425269 6342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:24.434731 6342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:37319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:24.435912 6342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:24.437122 6342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:24.448259 6351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:24.448305 6355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:24.448342 6352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:24.449152 6342 server_base.cc:1034] running on GCE node
I20260810 14:36:24.449812 6342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:24.451151 6342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:24.452344 6342 hybrid_clock.cc:648] HybridClock initialized: now 1786372584452305 us; error 29 us; skew 500 ppm
I20260810 14:36:24.452666 6342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:24 dist-test-slave-q9h9 krb5kdc[6310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372584, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:24.455585 6342 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:36:24.456944 6342 webserver.cc:460] Webserver started at http://127.6.46.254:44909/ using document root <none> and password file <none>
I20260810 14:36:24.457564 6342 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:24.457680 6342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:24.457975 6342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:24.460764 6342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "bd920e2537de41ad8c4616d20b2ab5b4"
format_stamp: "Formatted at 2026-08-10 14:36:24 on dist-test-slave-q9h9"
server_key: "74a485e7118e04fb01da26a11e3d59a1"
server_key_iv: "5e703f7de1f94bb0f1c234d432c9da09"
server_key_version: "encryptionkey@0"
I20260810 14:36:24.461467 6342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "bd920e2537de41ad8c4616d20b2ab5b4"
format_stamp: "Formatted at 2026-08-10 14:36:24 on dist-test-slave-q9h9"
server_key: "74a485e7118e04fb01da26a11e3d59a1"
server_key_iv: "5e703f7de1f94bb0f1c234d432c9da09"
server_key_version: "encryptionkey@0"
I20260810 14:36:24.466187 6342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260810 14:36:24.469285 6367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:24.470501 6342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:36:24.470657 6342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "bd920e2537de41ad8c4616d20b2ab5b4"
format_stamp: "Formatted at 2026-08-10 14:36:24 on dist-test-slave-q9h9"
server_key: "74a485e7118e04fb01da26a11e3d59a1"
server_key_iv: "5e703f7de1f94bb0f1c234d432c9da09"
server_key_version: "encryptionkey@0"
I20260810 14:36:24.470819 6342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:24.508351 6342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:24.511871 6342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:24.512125 6342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:24.528839 6342 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:37319
I20260810 14:36:24.528878 6441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:37319 every 8 connection(s)
I20260810 14:36:24.530072 6342 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:24.533427 6442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:24.537238 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 6342
I20260810 14:36:24.537411 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:24.537823 6331 external_mini_cluster.cc:1428] Setting key 5e8eafcd3ba42ed12bf00c8b3417738b
I20260810 14:36:24.543834 6442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4: Bootstrap starting.
I20260810 14:36:24.547199 6442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:24.548357 6442 log.cc:824] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:24.551507 6442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4: No bootstrap required, opened a new log
Aug 10 14:36:24 dist-test-slave-q9h9 krb5kdc[6310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:36:24.560312 6442 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd920e2537de41ad8c4616d20b2ab5b4" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 37319 } }
I20260810 14:36:24.560609 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:24.539467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.6.46.254:37319)
0810 14:36:24.540176 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:24.540186 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:24.540229 (+ 43us) server_negotiation.cc:407] Connection header received
0810 14:36:24.542481 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:24.542503 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:24.542858 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:24.543155 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:24.545128 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.546459 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.548174 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:24.548891 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:24.549097 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:24.553787 (+ 4690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:24.553798 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:24.553811 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:24.553845 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:24.556907 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:24.557659 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:24.557664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:24.557673 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:24.557832 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:24.558235 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:24.558238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:24.558240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:24.558733 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:24.558947 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:24.559357 (+ 410us) server_negotiation.cc:299] Negotiation successful
0810 14:36:24.559663 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":325,"threads_started":1}
I20260810 14:36:24.560652 6442 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:24.560748 6442 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd920e2537de41ad8c4616d20b2ab5b4, State: Initialized, Role: FOLLOWER
I20260810 14:36:24.561403 6442 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd920e2537de41ad8c4616d20b2ab5b4" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 37319 } }
I20260810 14:36:24.561690 6442 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:24.561830 6442 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:24.561997 6442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:24.564369 6442 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd920e2537de41ad8c4616d20b2ab5b4" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 37319 } }
I20260810 14:36:24.564813 6442 leader_election.cc:302] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd920e2537de41ad8c4616d20b2ab5b4; no voters:
I20260810 14:36:24.565320 6442 leader_election.cc:288] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:24.565426 6449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:24.566073 6449 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [term 1 LEADER]: Becoming Leader. State: Replica: bd920e2537de41ad8c4616d20b2ab5b4, State: Running, Role: LEADER
I20260810 14:36:24.566594 6449 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd920e2537de41ad8c4616d20b2ab5b4" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 37319 } }
I20260810 14:36:24.567102 6442 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:24.570367 6453 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd920e2537de41ad8c4616d20b2ab5b4. Latest consensus state: current_term: 1 leader_uuid: "bd920e2537de41ad8c4616d20b2ab5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd920e2537de41ad8c4616d20b2ab5b4" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 37319 } } }
I20260810 14:36:24.570574 6451 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd920e2537de41ad8c4616d20b2ab5b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd920e2537de41ad8c4616d20b2ab5b4" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 37319 } } }
I20260810 14:36:24.571164 6453 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:24.571215 6451 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:24.571915 6468 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:24.576663 6468 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:24.585680 6468 catalog_manager.cc:1347] Generated new cluster ID: 144f3f75303b411c9010345afe97b753
I20260810 14:36:24.585772 6468 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:24.600531 6468 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:24.602233 6468 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:24.614055 6468 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4: Generated new TSK 0
I20260810 14:36:24.614929 6468 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.46.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
Entry for principal kudu/127.6.46.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab.
I20260810 14:36:24.682901 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:37319
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:25.002601 6488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:25.002862 6488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:25.003002 6488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:25.009855 6488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:25.009961 6488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:25.010133 6488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:36:25.019404 6488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.193.keytab
--principal=kudu/127.6.46.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:37319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:25.020494 6488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:25.021689 6488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:25.033921 6493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:25.033880 6497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:25.033949 6494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:25.034736 6488 server_base.cc:1034] running on GCE node
I20260810 14:36:25.035231 6488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:25.036060 6488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:25.037261 6488 hybrid_clock.cc:648] HybridClock initialized: now 1786372585037210 us; error 41 us; skew 500 ppm
I20260810 14:36:25.037530 6488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:25 dist-test-slave-q9h9 krb5kdc[6310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372585, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:25.040369 6488 init.cc:377] Logged in from keytab as kudu/127.6.46.193@KRBTEST.COM (short username kudu)
I20260810 14:36:25.041628 6488 webserver.cc:460] Webserver started at http://127.6.46.193:37523/ using document root <none> and password file <none>
I20260810 14:36:25.042285 6488 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:25.042402 6488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:25.042696 6488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:25.045408 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "06f303f6116547bfa5d9421d5c91456b"
format_stamp: "Formatted at 2026-08-10 14:36:25 on dist-test-slave-q9h9"
server_key: "53c84f7bebb490db1bccb60ef8ff5841"
server_key_iv: "918b6db11242c109abe7df4ae3cded38"
server_key_version: "encryptionkey@0"
I20260810 14:36:25.046372 6488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "06f303f6116547bfa5d9421d5c91456b"
format_stamp: "Formatted at 2026-08-10 14:36:25 on dist-test-slave-q9h9"
server_key: "53c84f7bebb490db1bccb60ef8ff5841"
server_key_iv: "918b6db11242c109abe7df4ae3cded38"
server_key_version: "encryptionkey@0"
I20260810 14:36:25.051156 6488 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:36:25.054629 6505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:25.055812 6488 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260810 14:36:25.055958 6488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "06f303f6116547bfa5d9421d5c91456b"
format_stamp: "Formatted at 2026-08-10 14:36:25 on dist-test-slave-q9h9"
server_key: "53c84f7bebb490db1bccb60ef8ff5841"
server_key_iv: "918b6db11242c109abe7df4ae3cded38"
server_key_version: "encryptionkey@0"
I20260810 14:36:25.056121 6488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:25.091884 6488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:25.095943 6488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:25.096221 6488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:25.097112 6488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:25.098412 6488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:25.098495 6488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:25.098619 6488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:25.098678 6488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:25.128392 6488 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:44043
I20260810 14:36:25.128456 6692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:44043 every 8 connection(s)
I20260810 14:36:25.129565 6488 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:36:25.138166 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 6488
I20260810 14:36:25.138339 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:36:25.138890 6331 external_mini_cluster.cc:1428] Setting key 79e26551c19ebaf131e69c24d2d5726b
Aug 10 14:36:25 dist-test-slave-q9h9 krb5kdc[6310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372585, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.193@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:36:25.148878 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:25.131853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:40679 (local address 127.6.46.254:37319)
0810 14:36:25.132248 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:25.132253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:25.133066 (+ 813us) server_negotiation.cc:407] Connection header received
0810 14:36:25.134872 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:25.134877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:25.134955 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:25.135051 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:25.136463 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:25.137185 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:25.139145 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:25.139665 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:25.139773 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:25.143604 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:25.143609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:25.143612 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:25.143644 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:25.146080 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:25.146819 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:25.146822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:25.146824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:25.146946 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:25.147349 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:25.147352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:25.147354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:25.147614 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:25.147730 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:25.148471 (+ 741us) server_negotiation.cc:299] Negotiation successful
0810 14:36:25.148634 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":201,"threads_started":1}
I20260810 14:36:25.149978 6696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:25.132268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:37319 (local address 127.6.46.193:40679)
0810 14:36:25.132910 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:25.132953 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:25.134607 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:25.135285 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:25.135292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:25.135780 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:25.136295 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:25.136310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:25.137330 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:25.137333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:25.138986 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:25.139001 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:25.139904 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:25.139912 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:25.140176 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:25.140841 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:25.140869 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:25.143395 (+ 2526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:25.146202 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:25.146209 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:25.146222 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:25.146719 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:25.147034 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:25.147038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:25.147041 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:25.147258 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:25.147731 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:25.147737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:25.148105 (+ 368us) client_negotiation.cc:769] Sending connection context
0810 14:36:25.148365 (+ 260us) client_negotiation.cc:240] Negotiation successful
0810 14:36:25.148794 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":260,"threads_started":1}
I20260810 14:36:25.151389 6693 heartbeater.cc:347] Connected to a master server at 127.6.46.254:37319
I20260810 14:36:25.151677 6693 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:25.152434 6693 heartbeater.cc:511] Master 127.6.46.254:37319 requested a full tablet report, sending...
I20260810 14:36:25.154611 6390 ts_manager.cc:194] Registered new tserver with Master: 06f303f6116547bfa5d9421d5c91456b (127.6.46.193:44043)
I20260810 14:36:25.156032 6390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.193@KRBTEST.COM'} at 127.6.46.193:40679
WARNING: no policy specified for kudu/127.6.46.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
Entry for principal kudu/127.6.46.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab.
I20260810 14:36:25.198822 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:37319
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:25.514128 6700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:25.514381 6700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:25.514520 6700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:25.521574 6700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:25.521679 6700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:25.521824 6700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:36:25.532706 6700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.194.keytab
--principal=kudu/127.6.46.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:37319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:25.533995 6700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:25.535549 6700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:25.549031 6705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:25.549072 6707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:25.549167 6710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:25.549849 6700 server_base.cc:1034] running on GCE node
I20260810 14:36:25.550451 6700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:25.551257 6700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:25.552435 6700 hybrid_clock.cc:648] HybridClock initialized: now 1786372585552370 us; error 58 us; skew 500 ppm
I20260810 14:36:25.552714 6700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:25 dist-test-slave-q9h9 krb5kdc[6310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372585, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:25.555833 6700 init.cc:377] Logged in from keytab as kudu/127.6.46.194@KRBTEST.COM (short username kudu)
I20260810 14:36:25.557301 6700 webserver.cc:460] Webserver started at http://127.6.46.194:39069/ using document root <none> and password file <none>
I20260810 14:36:25.557888 6700 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:25.557986 6700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:25.558282 6700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:25.561031 6700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "1993ef87f04346c38702178aa632abc9"
format_stamp: "Formatted at 2026-08-10 14:36:25 on dist-test-slave-q9h9"
server_key: "d8f8f824dd4a3f722ab586fb1bfde6bc"
server_key_iv: "24a0b691b756cf30f782ac58f2984cce"
server_key_version: "encryptionkey@0"
I20260810 14:36:25.561741 6700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "1993ef87f04346c38702178aa632abc9"
format_stamp: "Formatted at 2026-08-10 14:36:25 on dist-test-slave-q9h9"
server_key: "d8f8f824dd4a3f722ab586fb1bfde6bc"
server_key_iv: "24a0b691b756cf30f782ac58f2984cce"
server_key_version: "encryptionkey@0"
I20260810 14:36:25.566370 6700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:36:25.569672 6720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:25.570960 6700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:25.571111 6700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "1993ef87f04346c38702178aa632abc9"
format_stamp: "Formatted at 2026-08-10 14:36:25 on dist-test-slave-q9h9"
server_key: "d8f8f824dd4a3f722ab586fb1bfde6bc"
server_key_iv: "24a0b691b756cf30f782ac58f2984cce"
server_key_version: "encryptionkey@0"
I20260810 14:36:25.571298 6700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:25.586652 6700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:25.590225 6700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:25.590474 6700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:25.591351 6700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:25.592561 6700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:25.592643 6700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:25.592773 6700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:25.592836 6700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:25.622771 6700 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:37705
I20260810 14:36:25.622805 6901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:37705 every 8 connection(s)
I20260810 14:36:25.623943 6700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:36:25.631333 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 6700
I20260810 14:36:25.631529 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:36:25.632088 6331 external_mini_cluster.cc:1428] Setting key f2d2d20ef7601558009facd131d7cc96
Aug 10 14:36:25 dist-test-slave-q9h9 krb5kdc[6310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372585, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.194@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:36:25.645080 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:25.626528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:46157 (local address 127.6.46.254:37319)
0810 14:36:25.626711 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:25.626716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:25.628082 (+ 1366us) server_negotiation.cc:407] Connection header received
0810 14:36:25.630638 (+ 2556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:25.630645 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:25.630752 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:25.630848 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:25.632634 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:25.633462 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:25.635477 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:25.635912 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:25.636009 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:25.639710 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:25.639715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:25.639718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:25.639750 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:25.642176 (+ 2426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:25.642949 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:25.642952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:25.642955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:25.643079 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:25.643494 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:25.643497 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:25.643499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:25.643736 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:25.643842 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:25.644522 (+ 680us) server_negotiation.cc:299] Negotiation successful
0810 14:36:25.644722 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:36:25.646131 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:25.627072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:37319 (local address 127.6.46.194:46157)
0810 14:36:25.627914 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:25.627971 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:25.630317 (+ 2346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:25.631086 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:25.631096 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:25.631808 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:25.632461 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:25.632491 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:25.633623 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:25.633628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:25.635339 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:25.635353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:25.636127 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:25.636134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:25.636406 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:25.637033 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:25.637058 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:25.639495 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:25.642318 (+ 2823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:25.642326 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:25.642339 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:25.642847 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:25.643174 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:25.643179 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:25.643183 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:25.643406 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:25.643844 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:25.643851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:25.644205 (+ 354us) client_negotiation.cc:769] Sending connection context
0810 14:36:25.644468 (+ 263us) client_negotiation.cc:240] Negotiation successful
0810 14:36:25.644913 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":327,"threads_started":1}
I20260810 14:36:25.647521 6902 heartbeater.cc:347] Connected to a master server at 127.6.46.254:37319
I20260810 14:36:25.647807 6902 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:25.648504 6902 heartbeater.cc:511] Master 127.6.46.254:37319 requested a full tablet report, sending...
I20260810 14:36:25.649825 6390 ts_manager.cc:194] Registered new tserver with Master: 1993ef87f04346c38702178aa632abc9 (127.6.46.194:37705)
I20260810 14:36:25.650614 6390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.194@KRBTEST.COM'} at 127.6.46.194:46157
WARNING: no policy specified for kudu/127.6.46.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
Entry for principal kudu/127.6.46.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab.
I20260810 14:36:25.694232 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:37319
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:34113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:26.012917 6909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:26.013195 6909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:26.013334 6909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:26.020330 6909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:26.020447 6909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:26.020591 6909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:36:26.029708 6909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:34113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.195.keytab
--principal=kudu/127.6.46.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:37319
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:26.030866 6909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:26.032085 6909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:26.044039 6916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:26.044131 6918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:26.044080 6922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:26.045199 6909 server_base.cc:1034] running on GCE node
I20260810 14:36:26.045728 6909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:26.046595 6909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:26.047794 6909 hybrid_clock.cc:648] HybridClock initialized: now 1786372586047743 us; error 39 us; skew 500 ppm
I20260810 14:36:26.048084 6909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:26 dist-test-slave-q9h9 krb5kdc[6310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372586, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:26.051324 6909 init.cc:377] Logged in from keytab as kudu/127.6.46.195@KRBTEST.COM (short username kudu)
I20260810 14:36:26.052639 6909 webserver.cc:460] Webserver started at http://127.6.46.195:39487/ using document root <none> and password file <none>
I20260810 14:36:26.053269 6909 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:26.053376 6909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:26.053664 6909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:26.056476 6909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "8b837954cdcb457d8182add017bbdffb"
format_stamp: "Formatted at 2026-08-10 14:36:26 on dist-test-slave-q9h9"
server_key: "16ded30de7da2598ee8bd5e13055485d"
server_key_iv: "91211d1358ce42046a27b8a231e96294"
server_key_version: "encryptionkey@0"
I20260810 14:36:26.057209 6909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "8b837954cdcb457d8182add017bbdffb"
format_stamp: "Formatted at 2026-08-10 14:36:26 on dist-test-slave-q9h9"
server_key: "16ded30de7da2598ee8bd5e13055485d"
server_key_iv: "91211d1358ce42046a27b8a231e96294"
server_key_version: "encryptionkey@0"
I20260810 14:36:26.061969 6909 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260810 14:36:26.065361 6933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:26.066618 6909 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:36:26.066761 6909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "8b837954cdcb457d8182add017bbdffb"
format_stamp: "Formatted at 2026-08-10 14:36:26 on dist-test-slave-q9h9"
server_key: "16ded30de7da2598ee8bd5e13055485d"
server_key_iv: "91211d1358ce42046a27b8a231e96294"
server_key_version: "encryptionkey@0"
I20260810 14:36:26.066926 6909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:26.076704 6909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:26.080070 6909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:26.080288 6909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:26.081070 6909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:26.082237 6909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:26.082306 6909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:26.082391 6909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:26.082432 6909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:26.111758 6909 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:43333
I20260810 14:36:26.111804 7133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:43333 every 8 connection(s)
I20260810 14:36:26.112933 6909 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:36:26.116519 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 6909
I20260810 14:36:26.116707 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:36:26.117156 6331 external_mini_cluster.cc:1428] Setting key 3cf4f927cdf00fb2c4a1ffcb1a7f6277
Aug 10 14:36:26 dist-test-slave-q9h9 krb5kdc[6310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372586, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.195@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:36:26.131846 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.115366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:44803 (local address 127.6.46.254:37319)
0810 14:36:26.115559 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:26.115564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:26.116535 (+ 971us) server_negotiation.cc:407] Connection header received
0810 14:36:26.118550 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:26.118555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:26.118644 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:26.118749 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:26.120264 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.121016 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.122470 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.122857 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.122946 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.126084 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:26.126089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:26.126092 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:26.126125 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:26.128680 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:26.129470 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:26.129474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:26.129476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:26.129599 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:26.130069 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:26.130073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:26.130076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:26.130362 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:26.130511 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:26.131297 (+ 786us) server_negotiation.cc:299] Negotiation successful
0810 14:36:26.131513 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 14:36:26.132839 7137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.115708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:37319 (local address 127.6.46.195:44803)
0810 14:36:26.116381 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:26.116416 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:26.118265 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:26.119000 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:26.119009 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:26.119484 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:26.120120 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:26.120138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.121116 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.121119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:26.122341 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:26.122358 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.123020 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.123024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:26.123198 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.123729 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:36:26.123748 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:26.125859 (+ 2111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:26.128818 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:26.128827 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:26.128840 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:26.129369 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:26.129709 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:36:26.129714 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:36:26.129718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:36:26.129943 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:36:26.130485 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:26.130492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:26.130872 (+ 380us) client_negotiation.cc:769] Sending connection context
0810 14:36:26.131160 (+ 288us) client_negotiation.cc:240] Negotiation successful
0810 14:36:26.131638 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":272,"threads_started":1}
I20260810 14:36:26.134075 7135 heartbeater.cc:347] Connected to a master server at 127.6.46.254:37319
I20260810 14:36:26.134343 7135 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:26.134989 7135 heartbeater.cc:511] Master 127.6.46.254:37319 requested a full tablet report, sending...
I20260810 14:36:26.136284 6390 ts_manager.cc:194] Registered new tserver with Master: 8b837954cdcb457d8182add017bbdffb (127.6.46.195:43333)
I20260810 14:36:26.137035 6390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.46.195@KRBTEST.COM'} at 127.6.46.195:44803
I20260810 14:36:26.143721 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:36:26.158564 6693 heartbeater.cc:503] Master 127.6.46.254:37319 was elected leader, sending a full tablet report...
I20260810 14:36:26.159215 6695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.147564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56430 (local address 127.6.46.254:37319)
0810 14:36:26.147716 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:26.147720 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:26.148200 (+ 480us) server_negotiation.cc:407] Connection header received
0810 14:36:26.148351 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:26.148357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:26.148432 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:26.148508 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:26.149853 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.150581 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.151809 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.152219 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.152324 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.154542 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:26.154548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:26.154551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:26.154579 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:26.156727 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:26.157325 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:26.157330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:26.157334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:26.157458 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:26.157799 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:26.157805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:26.157808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:26.158160 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:26.158380 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:26.158722 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 14:36:26.158916 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:36:26.162734 6389 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:36:26.169420 6389 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:36:26.204564 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.181974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.6.46.194:37705)
0810 14:36:26.193181 (+ 11207us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:26.193191 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:26.196102 (+ 2911us) server_negotiation.cc:407] Connection header received
0810 14:36:26.196589 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:26.196599 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:26.196869 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:26.197036 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:26.197961 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.199226 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.201904 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.203310 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.203463 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.203661 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:26.204082 (+ 421us) server_negotiation.cc:299] Negotiation successful
0810 14:36:26.204269 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11024,"thread_start_us":11201,"threads_started":1}
I20260810 14:36:26.204577 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.181167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:43333 (local address 127.0.0.1:45808)
0810 14:36:26.193892 (+ 12725us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:26.193932 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:26.194118 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:26.197930 (+ 3812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:26.197935 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:26.197959 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:26.198355 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:26.198368 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.200215 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.200219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:26.201941 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:26.201955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.203626 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.203633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:26.203817 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.203849 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 14:36:26.203941 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 14:36:26.204140 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12473,"mutex_wait_us":23,"thread_start_us":15060,"threads_started":1}
I20260810 14:36:26.205381 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.182594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:44043 (local address 127.0.0.1:41964)
0810 14:36:26.196578 (+ 13984us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:26.196594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:26.196972 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:26.199219 (+ 2247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:26.199223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:26.199249 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:26.199685 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:26.199697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.201579 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.201585 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:26.203021 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:26.203034 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.204709 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.204717 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:26.204881 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.204912 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 14:36:26.204973 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 14:36:26.205082 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13858,"thread_start_us":13700,"threads_started":1}
I20260810 14:36:26.205636 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.181209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.6.46.195:43333)
0810 14:36:26.197496 (+ 16287us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:26.197502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:26.197528 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:36:26.197597 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:26.197601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:26.197757 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:26.197915 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:26.198590 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.200063 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.202108 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.203390 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.205015 (+ 1625us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.205187 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:26.205237 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 14:36:26.205340 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16145,"thread_start_us":16010,"threads_started":1}
I20260810 14:36:26.205953 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.183253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41964 (local address 127.6.46.193:44043)
0810 14:36:26.198672 (+ 15419us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:26.198679 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:26.198706 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:36:26.198810 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:26.198818 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:26.199024 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:26.199226 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:26.199878 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.201431 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.203174 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.204448 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.205180 (+ 732us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.205397 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:26.205474 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 14:36:26.205620 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15258,"thread_start_us":15085,"threads_started":1}
I20260810 14:36:26.206971 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.181444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:37705 (local address 127.0.0.1:42640)
0810 14:36:26.195958 (+ 14514us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:26.195977 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:26.196115 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:26.197356 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:26.197364 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:26.197397 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:26.197775 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:26.197786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.199385 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.199391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:26.201761 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:26.201776 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.203517 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.203524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:26.203748 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.203835 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 14:36:26.203960 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 14:36:26.206608 (+ 2648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14386,"mutex_wait_us":1653,"thread_start_us":12055,"threads_started":1}
I20260810 14:36:26.209290 7019 tablet_service.cc:1467] Processing CreateTablet for tablet b652028e56f347d6a20dab0bb627656f (DEFAULT_TABLE table=test-table [id=cc249ea01c0045599c568ae4a524b38a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:26.209280 6578 tablet_service.cc:1467] Processing CreateTablet for tablet b652028e56f347d6a20dab0bb627656f (DEFAULT_TABLE table=test-table [id=cc249ea01c0045599c568ae4a524b38a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:26.210614 7019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b652028e56f347d6a20dab0bb627656f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:26.210215 6804 tablet_service.cc:1467] Processing CreateTablet for tablet b652028e56f347d6a20dab0bb627656f (DEFAULT_TABLE table=test-table [id=cc249ea01c0045599c568ae4a524b38a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:26.210726 6578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b652028e56f347d6a20dab0bb627656f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:26.211447 6804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b652028e56f347d6a20dab0bb627656f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:26.221180 7160 tablet_bootstrap.cc:492] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb: Bootstrap starting.
I20260810 14:36:26.224220 7161 tablet_bootstrap.cc:492] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b: Bootstrap starting.
I20260810 14:36:26.225432 7160 tablet_bootstrap.cc:654] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:26.226661 7167 tablet_bootstrap.cc:492] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9: Bootstrap starting.
I20260810 14:36:26.226765 7160 log.cc:824] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:26.227674 7161 tablet_bootstrap.cc:654] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:26.228924 7161 log.cc:824] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:26.229339 7167 tablet_bootstrap.cc:654] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:26.230743 7167 log.cc:824] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:26.231778 7160 tablet_bootstrap.cc:492] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb: No bootstrap required, opened a new log
I20260810 14:36:26.231776 7161 tablet_bootstrap.cc:492] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b: No bootstrap required, opened a new log
I20260810 14:36:26.232043 7160 ts_tablet_manager.cc:1397] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260810 14:36:26.232043 7161 ts_tablet_manager.cc:1397] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 14:36:26.233059 7167 tablet_bootstrap.cc:492] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9: No bootstrap required, opened a new log
I20260810 14:36:26.233275 7167 ts_tablet_manager.cc:1397] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 14:36:26.240808 7167 raft_consensus.cc:348] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b837954cdcb457d8182add017bbdffb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43333 } } peers { permanent_uuid: "1993ef87f04346c38702178aa632abc9" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37705 } } peers { permanent_uuid: "06f303f6116547bfa5d9421d5c91456b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44043 } }
I20260810 14:36:26.241159 7167 raft_consensus.cc:374] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:26.241281 7167 raft_consensus.cc:729] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1993ef87f04346c38702178aa632abc9, State: Initialized, Role: FOLLOWER
I20260810 14:36:26.241814 7167 consensus_queue.cc:260] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b837954cdcb457d8182add017bbdffb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43333 } } peers { permanent_uuid: "1993ef87f04346c38702178aa632abc9" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37705 } } peers { permanent_uuid: "06f303f6116547bfa5d9421d5c91456b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44043 } }
I20260810 14:36:26.242339 7160 raft_consensus.cc:348] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b837954cdcb457d8182add017bbdffb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43333 } } peers { permanent_uuid: "1993ef87f04346c38702178aa632abc9" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37705 } } peers { permanent_uuid: "06f303f6116547bfa5d9421d5c91456b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44043 } }
I20260810 14:36:26.242549 7161 raft_consensus.cc:348] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b837954cdcb457d8182add017bbdffb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43333 } } peers { permanent_uuid: "1993ef87f04346c38702178aa632abc9" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37705 } } peers { permanent_uuid: "06f303f6116547bfa5d9421d5c91456b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44043 } }
I20260810 14:36:26.242816 7160 raft_consensus.cc:374] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:26.242835 6902 heartbeater.cc:503] Master 127.6.46.254:37319 was elected leader, sending a full tablet report...
I20260810 14:36:26.242908 7160 raft_consensus.cc:729] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b837954cdcb457d8182add017bbdffb, State: Initialized, Role: FOLLOWER
I20260810 14:36:26.243036 7161 raft_consensus.cc:374] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:26.243170 7161 raft_consensus.cc:729] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f303f6116547bfa5d9421d5c91456b, State: Initialized, Role: FOLLOWER
I20260810 14:36:26.243495 7167 ts_tablet_manager.cc:1428] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260810 14:36:26.243603 7160 consensus_queue.cc:260] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b837954cdcb457d8182add017bbdffb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43333 } } peers { permanent_uuid: "1993ef87f04346c38702178aa632abc9" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37705 } } peers { permanent_uuid: "06f303f6116547bfa5d9421d5c91456b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44043 } }
I20260810 14:36:26.243820 7161 consensus_queue.cc:260] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b837954cdcb457d8182add017bbdffb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43333 } } peers { permanent_uuid: "1993ef87f04346c38702178aa632abc9" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37705 } } peers { permanent_uuid: "06f303f6116547bfa5d9421d5c91456b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44043 } }
I20260810 14:36:26.244802 7135 heartbeater.cc:503] Master 127.6.46.254:37319 was elected leader, sending a full tablet report...
I20260810 14:36:26.245436 7160 ts_tablet_manager.cc:1428] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260810 14:36:26.247913 7161 ts_tablet_manager.cc:1428] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
W20260810 14:36:26.364446 7136 tablet.cc:2369] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:36:26.375769 6903 tablet.cc:2369] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:36:26.382112 6694 tablet.cc:2369] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:36:26.607889 7173 raft_consensus.cc:482] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:36:26.608188 7173 raft_consensus.cc:504] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b837954cdcb457d8182add017bbdffb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43333 } } peers { permanent_uuid: "1993ef87f04346c38702178aa632abc9" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37705 } } peers { permanent_uuid: "06f303f6116547bfa5d9421d5c91456b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44043 } }
I20260810 14:36:26.609870 7173 leader_election.cc:288] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b837954cdcb457d8182add017bbdffb (127.6.46.195:43333), 06f303f6116547bfa5d9421d5c91456b (127.6.46.193:44043)
I20260810 14:36:26.616400 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.610555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:41863 (local address 127.6.46.193:44043)
0810 14:36:26.610772 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:26.610778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:26.611428 (+ 650us) server_negotiation.cc:407] Connection header received
0810 14:36:26.611592 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:26.611596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:26.611654 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:26.611725 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:26.612343 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.613050 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.614782 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.615513 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.615656 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.615730 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:26.616034 (+ 304us) server_negotiation.cc:299] Negotiation successful
0810 14:36:26.616156 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 14:36:26.616443 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.610345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:42317 (local address 127.6.46.195:43333)
0810 14:36:26.610568 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:26.610575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:26.611258 (+ 683us) server_negotiation.cc:407] Connection header received
0810 14:36:26.611462 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:26.611467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:26.611527 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:26.611605 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:26.612191 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.612882 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.614674 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.615566 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.615711 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.615784 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:26.616070 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 14:36:26.616201 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 14:36:26.616547 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.610430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:44043 (local address 127.6.46.194:41863)
0810 14:36:26.611322 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:26.611336 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:26.611461 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:26.611764 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:26.611769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:26.611812 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:26.612201 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:26.612209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.613200 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.613204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:26.614621 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:26.614633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.615737 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.615743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:26.615880 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.615903 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:36:26.615981 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 14:36:26.616096 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":1,"thread_start_us":574,"threads_started":1}
I20260810 14:36:26.616570 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.610247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:43333 (local address 127.6.46.194:42317)
0810 14:36:26.611149 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:26.611164 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:26.611318 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:26.611635 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:26.611639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:26.611673 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:26.612030 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:26.612040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.613015 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.613021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:26.614510 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:26.614522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.615783 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.615790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:26.615918 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.615941 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:36:26.616021 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 14:36:26.616136 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":781,"mutex_wait_us":4,"thread_start_us":518,"threads_started":1}
I20260810 14:36:26.617633 6611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b652028e56f347d6a20dab0bb627656f" candidate_uuid: "1993ef87f04346c38702178aa632abc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f303f6116547bfa5d9421d5c91456b" is_pre_election: true
I20260810 14:36:26.617872 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b652028e56f347d6a20dab0bb627656f" candidate_uuid: "1993ef87f04346c38702178aa632abc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b837954cdcb457d8182add017bbdffb" is_pre_election: true
I20260810 14:36:26.618153 6611 raft_consensus.cc:2454] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1993ef87f04346c38702178aa632abc9 in term 0.
I20260810 14:36:26.618362 7061 raft_consensus.cc:2454] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1993ef87f04346c38702178aa632abc9 in term 0.
I20260810 14:36:26.618798 6721 leader_election.cc:302] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06f303f6116547bfa5d9421d5c91456b, 1993ef87f04346c38702178aa632abc9; no voters:
I20260810 14:36:26.619176 7173 raft_consensus.cc:2781] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:36:26.619330 7173 raft_consensus.cc:482] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:36:26.619441 7173 raft_consensus.cc:3037] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:26.621517 7173 raft_consensus.cc:504] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b837954cdcb457d8182add017bbdffb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43333 } } peers { permanent_uuid: "1993ef87f04346c38702178aa632abc9" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37705 } } peers { permanent_uuid: "06f303f6116547bfa5d9421d5c91456b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44043 } }
I20260810 14:36:26.622123 7173 leader_election.cc:288] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [CANDIDATE]: Term 1 election: Requested vote from peers 8b837954cdcb457d8182add017bbdffb (127.6.46.195:43333), 06f303f6116547bfa5d9421d5c91456b (127.6.46.193:44043)
I20260810 14:36:26.622515 7061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b652028e56f347d6a20dab0bb627656f" candidate_uuid: "1993ef87f04346c38702178aa632abc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b837954cdcb457d8182add017bbdffb"
I20260810 14:36:26.622617 6611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b652028e56f347d6a20dab0bb627656f" candidate_uuid: "1993ef87f04346c38702178aa632abc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06f303f6116547bfa5d9421d5c91456b"
I20260810 14:36:26.622699 7061 raft_consensus.cc:3037] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:26.622802 6611 raft_consensus.cc:3037] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:26.624810 7061 raft_consensus.cc:2454] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1993ef87f04346c38702178aa632abc9 in term 1.
I20260810 14:36:26.625180 6723 leader_election.cc:302] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1993ef87f04346c38702178aa632abc9, 8b837954cdcb457d8182add017bbdffb; no voters:
I20260810 14:36:26.625473 7173 raft_consensus.cc:2781] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:26.625860 6611 raft_consensus.cc:2454] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1993ef87f04346c38702178aa632abc9 in term 1.
I20260810 14:36:26.625931 7173 raft_consensus.cc:686] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [term 1 LEADER]: Becoming Leader. State: Replica: 1993ef87f04346c38702178aa632abc9, State: Running, Role: LEADER
I20260810 14:36:26.626703 7173 consensus_queue.cc:237] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b837954cdcb457d8182add017bbdffb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43333 } } peers { permanent_uuid: "1993ef87f04346c38702178aa632abc9" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37705 } } peers { permanent_uuid: "06f303f6116547bfa5d9421d5c91456b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44043 } }
I20260810 14:36:26.630993 6388 catalog_manager.cc:5495] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1993ef87f04346c38702178aa632abc9 (127.6.46.194). New cstate: current_term: 1 leader_uuid: "1993ef87f04346c38702178aa632abc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b837954cdcb457d8182add017bbdffb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1993ef87f04346c38702178aa632abc9" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 37705 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06f303f6116547bfa5d9421d5c91456b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44043 } health_report { overall_health: UNKNOWN } } }
I20260810 14:36:26.661141 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.656049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.6.46.194:37705)
0810 14:36:26.656240 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:26.656244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:26.656353 (+ 109us) server_negotiation.cc:407] Connection header received
0810 14:36:26.656557 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:26.656561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:26.656635 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:26.656761 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:36:26.657120 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.657845 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.659208 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.659669 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.659757 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.660177 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:36:26.660671 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:36:26.660779 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:26.660857 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 14:36:26.660921 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 14:36:26.668771 6611 raft_consensus.cc:1261] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b [term 1 FOLLOWER]: Refusing update from remote peer 1993ef87f04346c38702178aa632abc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:26.669061 7061 raft_consensus.cc:1261] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb [term 1 FOLLOWER]: Refusing update from remote peer 1993ef87f04346c38702178aa632abc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:26.669637 7173 consensus_queue.cc:1035] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06f303f6116547bfa5d9421d5c91456b" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 44043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:26.669898 7179 consensus_queue.cc:1035] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b837954cdcb457d8182add017bbdffb" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 43333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:26.681198 7183 mvcc.cc:205] Tried to move back new op lower bound from 7316982114986024960 to 7316982114827059200. Current Snapshot: MvccSnapshot[applied={T|T < 7316982114986024960}]
I20260810 14:36:26.694664 6388 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:26.694904 6388 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:26.697955 6388 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bd920e2537de41ad8c4616d20b2ab5b4: Sending DeleteTablet for 3 replicas of tablet b652028e56f347d6a20dab0bb627656f
I20260810 14:36:26.698969 7019 tablet_service.cc:1514] Processing DeleteTablet for tablet b652028e56f347d6a20dab0bb627656f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:26 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:45808
I20260810 14:36:26.699009 6578 tablet_service.cc:1514] Processing DeleteTablet for tablet b652028e56f347d6a20dab0bb627656f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:26 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:41964
I20260810 14:36:26.699039 6804 tablet_service.cc:1514] Processing DeleteTablet for tablet b652028e56f347d6a20dab0bb627656f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:26 UTC) from {username='kudu', principal='kudu/127.6.46.254@KRBTEST.COM'} at 127.0.0.1:42640
I20260810 14:36:26.700929 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 6488
I20260810 14:36:26.714221 7194 tablet_replica.cc:330] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb: stopping tablet replica
I20260810 14:36:26.714637 7194 raft_consensus.cc:2229] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:26.715129 7194 raft_consensus.cc:2258] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:26.715855 7195 tablet_replica.cc:330] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b: stopping tablet replica
I20260810 14:36:26.716231 7195 raft_consensus.cc:2229] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:26.716692 7195 raft_consensus.cc:2258] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:26.718847 7194 ts_tablet_manager.cc:1905] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:26.725090 7195 ts_tablet_manager.cc:1905] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:26.725512 7194 ts_tablet_manager.cc:1918] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:26.725695 7194 log.cc:1196] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/b652028e56f347d6a20dab0bb627656f
I20260810 14:36:26.726258 7194 ts_tablet_manager.cc:1939] T b652028e56f347d6a20dab0bb627656f P 8b837954cdcb457d8182add017bbdffb: Deleting consensus metadata
I20260810 14:36:26.727913 6372 catalog_manager.cc:4841] TS 8b837954cdcb457d8182add017bbdffb (127.6.46.195:43333): tablet b652028e56f347d6a20dab0bb627656f (table test-table [id=cc249ea01c0045599c568ae4a524b38a]) successfully deleted
I20260810 14:36:26.730613 7196 tablet_replica.cc:330] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9: stopping tablet replica
I20260810 14:36:26.731019 7196 raft_consensus.cc:2229] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:36:26.731735 7195 ts_tablet_manager.cc:1918] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:26.732007 7196 raft_consensus.cc:2258] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:26.732131 7195 log.cc:1196] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/b652028e56f347d6a20dab0bb627656f
I20260810 14:36:26.733089 7195 ts_tablet_manager.cc:1939] T b652028e56f347d6a20dab0bb627656f P 06f303f6116547bfa5d9421d5c91456b: Deleting consensus metadata
I20260810 14:36:26.733692 7196 ts_tablet_manager.cc:1905] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:26.734767 6369 catalog_manager.cc:4841] TS 06f303f6116547bfa5d9421d5c91456b (127.6.46.193:44043): tablet b652028e56f347d6a20dab0bb627656f (table test-table [id=cc249ea01c0045599c568ae4a524b38a]) successfully deleted
Aug 10 14:36:26 dist-test-slave-q9h9 krb5kdc[6310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.193@KRBTEST.COM
I20260810 14:36:26.739645 7196 ts_tablet_manager.cc:1918] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:26.739794 7196 log.cc:1196] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/b652028e56f347d6a20dab0bb627656f
I20260810 14:36:26.740288 7196 ts_tablet_manager.cc:1939] T b652028e56f347d6a20dab0bb627656f P 1993ef87f04346c38702178aa632abc9: Deleting consensus metadata
I20260810 14:36:26.741820 6370 catalog_manager.cc:4841] TS 1993ef87f04346c38702178aa632abc9 (127.6.46.194:37705): tablet b652028e56f347d6a20dab0bb627656f (table test-table [id=cc249ea01c0045599c568ae4a524b38a]) successfully deleted
I20260810 14:36:26.743112 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.702337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.6.46.193:44043)
0810 14:36:26.702514 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:26.702521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:26.718461 (+ 15940us) server_negotiation.cc:407] Connection header received
0810 14:36:26.718871 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:26.718878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:26.718959 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:26.719031 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:26.720736 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.721833 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.730144 (+ 8311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.730743 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.730865 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.736367 (+ 5502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:26.736383 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:26.736399 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:26.736443 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:26.740020 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:26.740844 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:26.740854 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:26.740858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:26.741024 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:26.741457 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:26.741463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:26.741467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:26.741900 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:26.742232 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:26.742515 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 14:36:26.742725 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:36:26.743840 6686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:26.832286 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 6488
I20260810 14:36:26.856057 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 6700
Aug 10 14:36:26 dist-test-slave-q9h9 krb5kdc[6310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.194@KRBTEST.COM
I20260810 14:36:26.872069 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:26.856794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42664 (local address 127.6.46.194:37705)
0810 14:36:26.856974 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:26.856981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:26.857059 (+ 78us) server_negotiation.cc:407] Connection header received
0810 14:36:26.857256 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:26.857259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:26.857334 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:26.857423 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:26.858738 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.859537 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.860906 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:26.861351 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:26.861467 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:26.865995 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:26.866009 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:26.866057 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:26.866097 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:26.869164 (+ 3067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:26.869955 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:26.869962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:26.869964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:26.870273 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:26.870658 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:26.870662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:26.870663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:26.871018 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:26.871323 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:26.871581 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 14:36:26.871751 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 14:36:26.872839 6895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:26.982694 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 6700
I20260810 14:36:27.006381 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 6909
Aug 10 14:36:27 dist-test-slave-q9h9 krb5kdc[6310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.195@KRBTEST.COM
I20260810 14:36:27.020890 7151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:27.007123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.6.46.195:43333)
0810 14:36:27.007309 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:27.007315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:27.007335 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:36:27.007505 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:27.007510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:27.007586 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:27.007693 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:27.009018 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:27.009737 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:27.011069 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:27.011464 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:27.011576 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:27.015692 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:27.015706 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:27.015722 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:27.015757 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:27.018313 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:27.018931 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:27.018938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:27.018940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:27.019068 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:27.019438 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:27.019442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:27.019444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:27.019808 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:27.020080 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:27.020379 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 14:36:27.020583 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:36:27.021487 7129 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:27.107300 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 6909
I20260810 14:36:27.127352 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 6342
I20260810 14:36:27.128396 6437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:27.236377 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 6342
2026-08-10T14:36:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4627 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T14:36:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:27Z Disabled control of system clock
I20260810 14:36:27.321812 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43161
--webserver_interface=127.6.46.254
--webserver_port=0
--builtin_ntp_servers=127.6.46.212:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:43161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:36:27.641633 7221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:27.641889 7221 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:27.641978 7221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:27.649024 7221 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:27.649118 7221 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:27.649163 7221 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:27.649219 7221 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:27.649268 7221 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:27.658147 7221 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:43161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:27.659344 7221 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:27.661008 7221 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:27.672003 7230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:27.671954 7232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:27.672065 7235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:27.672756 7221 server_base.cc:1034] running on GCE node
I20260810 14:36:27.673409 7221 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:27.674677 7221 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:27.675884 7221 hybrid_clock.cc:648] HybridClock initialized: now 1786372587675834 us; error 35 us; skew 500 ppm
I20260810 14:36:27.676203 7221 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:36:27.680974 7221 webserver.cc:460] Webserver started at http://127.6.46.254:41201/ using document root <none> and password file <none>
I20260810 14:36:27.681599 7221 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:27.681715 7221 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:27.682030 7221 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:27.684737 7221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "787867886369437596d5f1bc6def73e9"
format_stamp: "Formatted at 2026-08-10 14:36:27 on dist-test-slave-q9h9"
server_key: "c1c6b167840aedabc623eb556c7f973b"
server_key_iv: "e1819c0fc9c4ef6c2d8acdfa604e6434"
server_key_version: "encryptionkey@0"
I20260810 14:36:27.685514 7221 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "787867886369437596d5f1bc6def73e9"
format_stamp: "Formatted at 2026-08-10 14:36:27 on dist-test-slave-q9h9"
server_key: "c1c6b167840aedabc623eb556c7f973b"
server_key_iv: "e1819c0fc9c4ef6c2d8acdfa604e6434"
server_key_version: "encryptionkey@0"
I20260810 14:36:27.690241 7221 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 14:36:27.693302 7241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:27.694537 7221 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:36:27.694690 7221 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "787867886369437596d5f1bc6def73e9"
format_stamp: "Formatted at 2026-08-10 14:36:27 on dist-test-slave-q9h9"
server_key: "c1c6b167840aedabc623eb556c7f973b"
server_key_iv: "e1819c0fc9c4ef6c2d8acdfa604e6434"
server_key_version: "encryptionkey@0"
I20260810 14:36:27.694890 7221 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:27.720085 7221 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:27.720853 7221 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:27.721098 7221 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:27.738149 7221 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:43161
I20260810 14:36:27.738184 7319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:43161 every 8 connection(s)
I20260810 14:36:27.739357 7221 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:27.742467 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 7221
I20260810 14:36:27.742626 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:27.742811 7320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:27.743084 6331 external_mini_cluster.cc:1428] Setting key ebec9b4dae20c781ec09c17f4655bd11
I20260810 14:36:27.757283 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9: Bootstrap starting.
I20260810 14:36:27.757658 7323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:27.744899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57482 (local address 127.6.46.254:43161)
0810 14:36:27.746038 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:27.746058 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:27.746108 (+ 50us) server_negotiation.cc:407] Connection header received
0810 14:36:27.747808 (+ 1700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:27.747834 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:27.748395 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:27.748838 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:27.749336 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:27.751337 (+ 2001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:27.752628 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:27.753658 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:27.753837 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:27.755332 (+ 1495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:27.755343 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:27.755358 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:27.756034 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:27.756134 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:27.756166 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:27.756274 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:27.756474 (+ 200us) server_negotiation.cc:299] Negotiation successful
0810 14:36:27.756754 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":337,"threads_started":1}
I20260810 14:36:27.760334 7320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:27.761251 7320 log.cc:824] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:27.763665 7320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9: No bootstrap required, opened a new log
I20260810 14:36:27.770433 7320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787867886369437596d5f1bc6def73e9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43161 } }
I20260810 14:36:27.770746 7320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:27.770830 7320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787867886369437596d5f1bc6def73e9, State: Initialized, Role: FOLLOWER
I20260810 14:36:27.771299 7320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787867886369437596d5f1bc6def73e9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43161 } }
I20260810 14:36:27.771517 7320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:27.771633 7320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:27.771785 7320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:27.773761 7320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787867886369437596d5f1bc6def73e9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43161 } }
I20260810 14:36:27.774284 7320 leader_election.cc:302] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 787867886369437596d5f1bc6def73e9; no voters:
I20260810 14:36:27.774874 7320 leader_election.cc:288] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:27.774991 7327 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:27.775699 7327 raft_consensus.cc:686] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [term 1 LEADER]: Becoming Leader. State: Replica: 787867886369437596d5f1bc6def73e9, State: Running, Role: LEADER
I20260810 14:36:27.776194 7327 consensus_queue.cc:237] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787867886369437596d5f1bc6def73e9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43161 } }
I20260810 14:36:27.776706 7320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:27.778790 7328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "787867886369437596d5f1bc6def73e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787867886369437596d5f1bc6def73e9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43161 } } }
I20260810 14:36:27.779227 7328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:27.779565 7329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 787867886369437596d5f1bc6def73e9. Latest consensus state: current_term: 1 leader_uuid: "787867886369437596d5f1bc6def73e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "787867886369437596d5f1bc6def73e9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43161 } } }
I20260810 14:36:27.779872 7329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:27.780508 7336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:27.785462 7336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:27.793187 7336 catalog_manager.cc:1347] Generated new cluster ID: 9858f0aba457434c850e34ea261c5abc
I20260810 14:36:27.793282 7336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:27.812270 7336 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:27.813422 7336 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:27.822468 7336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9: Generated new TSK 0
I20260810 14:36:27.823195 7336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:27.831954 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.193:0
--local_ip_for_outbound_sockets=127.6.46.193
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43161
--builtin_ntp_servers=127.6.46.212:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 14:36:28.156067 7354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:28.156356 7354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:28.156450 7354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:28.163301 7354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:28.163383 7354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:28.163483 7354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.193
I20260810 14:36:28.171916 7354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.46.193
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.193
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:28.173007 7354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:28.174638 7354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:28.186882 7363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:28.186925 7372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:28.186949 7367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:28.187675 7354 server_base.cc:1034] running on GCE node
I20260810 14:36:28.188243 7354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:28.189179 7354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:28.190354 7354 hybrid_clock.cc:648] HybridClock initialized: now 1786372588190304 us; error 47 us; skew 500 ppm
I20260810 14:36:28.190632 7354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:36:28.192634 7354 webserver.cc:460] Webserver started at http://127.6.46.193:39589/ using document root <none> and password file <none>
I20260810 14:36:28.193270 7354 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:28.193356 7354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:28.193625 7354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:28.196274 7354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/data/instance:
uuid: "bf2a475d0b4c42b6b67075b57ccbc46a"
format_stamp: "Formatted at 2026-08-10 14:36:28 on dist-test-slave-q9h9"
server_key: "cb0dfd537c3e84c9e503bcdd4cd9eb16"
server_key_iv: "6290af33b494c707983f2000e9dd16ed"
server_key_version: "encryptionkey@0"
I20260810 14:36:28.196973 7354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance:
uuid: "bf2a475d0b4c42b6b67075b57ccbc46a"
format_stamp: "Formatted at 2026-08-10 14:36:28 on dist-test-slave-q9h9"
server_key: "cb0dfd537c3e84c9e503bcdd4cd9eb16"
server_key_iv: "6290af33b494c707983f2000e9dd16ed"
server_key_version: "encryptionkey@0"
I20260810 14:36:28.201723 7354 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:36:28.204947 7386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:28.206264 7354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:28.206431 7354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/wal
uuid: "bf2a475d0b4c42b6b67075b57ccbc46a"
format_stamp: "Formatted at 2026-08-10 14:36:28 on dist-test-slave-q9h9"
server_key: "cb0dfd537c3e84c9e503bcdd4cd9eb16"
server_key_iv: "6290af33b494c707983f2000e9dd16ed"
server_key_version: "encryptionkey@0"
I20260810 14:36:28.206621 7354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:28.226938 7354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:28.227603 7354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:28.227851 7354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:28.228694 7354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:28.230048 7354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:28.230131 7354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:28.230213 7354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:28.230268 7354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:28.259460 7354 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.193:41955
I20260810 14:36:28.259545 7579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.193:41955 every 8 connection(s)
I20260810 14:36:28.260627 7354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/data/info.pb
I20260810 14:36:28.270299 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 7354
I20260810 14:36:28.270457 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/wal/instance
I20260810 14:36:28.270936 6331 external_mini_cluster.cc:1428] Setting key e127d7795614aee3cf2996f766f3c13c
I20260810 14:36:28.272265 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:28.263083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.193:53091 (local address 127.6.46.254:43161)
0810 14:36:28.263525 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:28.263530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:28.264654 (+ 1124us) server_negotiation.cc:407] Connection header received
0810 14:36:28.266332 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:28.266336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:28.266403 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:28.266501 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:28.267405 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:28.268117 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:28.269546 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:28.269933 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:28.270076 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:28.271451 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:28.271456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:28.271459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:28.271476 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:28.271522 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:28.271535 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:28.271612 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:28.271831 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 14:36:28.271923 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":231,"threads_started":1}
I20260810 14:36:28.273267 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:28.263636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:43161 (local address 127.6.46.193:53091)
0810 14:36:28.264493 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:28.264526 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:28.266111 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:28.266671 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:28.266678 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:28.266756 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:28.267280 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:28.267295 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:28.268255 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:28.268259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:28.269441 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:28.269452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:28.270176 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:28.270184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:28.270430 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:28.271155 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:28.271158 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:28.271194 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:28.271268 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:28.271625 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:28.271638 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:28.271645 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:36:28.271803 (+ 158us) client_negotiation.cc:240] Negotiation successful
0810 14:36:28.272152 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":264,"threads_started":1}
I20260810 14:36:28.273600 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.194:0
--local_ip_for_outbound_sockets=127.6.46.194
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43161
--builtin_ntp_servers=127.6.46.212:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:36:28.274785 7580 heartbeater.cc:347] Connected to a master server at 127.6.46.254:43161
I20260810 14:36:28.275045 7580 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:28.275717 7580 heartbeater.cc:511] Master 127.6.46.254:43161 requested a full tablet report, sending...
I20260810 14:36:28.277745 7268 ts_manager.cc:194] Registered new tserver with Master: bf2a475d0b4c42b6b67075b57ccbc46a (127.6.46.193:41955)
I20260810 14:36:28.279567 7268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.46.193:53091
W20260810 14:36:28.594560 7586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:28.594832 7586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:28.594956 7586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:28.601905 7586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:28.602001 7586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:28.602162 7586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.194
I20260810 14:36:28.610715 7586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.46.194
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.194
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:28.611836 7586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:28.613474 7586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:28.625587 7595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:28.625694 7602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:28.625701 7594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:28.626740 7586 server_base.cc:1034] running on GCE node
I20260810 14:36:28.627269 7586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:28.628100 7586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:28.629281 7586 hybrid_clock.cc:648] HybridClock initialized: now 1786372588629235 us; error 46 us; skew 500 ppm
I20260810 14:36:28.629551 7586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:36:28.631692 7586 webserver.cc:460] Webserver started at http://127.6.46.194:39543/ using document root <none> and password file <none>
I20260810 14:36:28.632547 7586 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:28.632659 7586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:28.632936 7586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:28.635574 7586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/data/instance:
uuid: "5dead38db0cf4b06a4a30c8f91a2478a"
format_stamp: "Formatted at 2026-08-10 14:36:28 on dist-test-slave-q9h9"
server_key: "927fb6f6e1ff40af83e8d334aa0d7165"
server_key_iv: "60375440791a41b122c89b7db82d8d53"
server_key_version: "encryptionkey@0"
I20260810 14:36:28.636272 7586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance:
uuid: "5dead38db0cf4b06a4a30c8f91a2478a"
format_stamp: "Formatted at 2026-08-10 14:36:28 on dist-test-slave-q9h9"
server_key: "927fb6f6e1ff40af83e8d334aa0d7165"
server_key_iv: "60375440791a41b122c89b7db82d8d53"
server_key_version: "encryptionkey@0"
I20260810 14:36:28.640856 7586 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 14:36:28.644419 7615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:28.645596 7586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 14:36:28.645746 7586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/wal
uuid: "5dead38db0cf4b06a4a30c8f91a2478a"
format_stamp: "Formatted at 2026-08-10 14:36:28 on dist-test-slave-q9h9"
server_key: "927fb6f6e1ff40af83e8d334aa0d7165"
server_key_iv: "60375440791a41b122c89b7db82d8d53"
server_key_version: "encryptionkey@0"
I20260810 14:36:28.645934 7586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:28.673408 7586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:28.674253 7586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:28.674531 7586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:28.675499 7586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:28.676920 7586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:28.677026 7586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:28.677150 7586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:28.677211 7586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:28.707070 7586 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.194:36619
I20260810 14:36:28.707129 7814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.194:36619 every 8 connection(s)
I20260810 14:36:28.708271 7586 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/data/info.pb
I20260810 14:36:28.712612 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 7586
I20260810 14:36:28.712803 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/wal/instance
I20260810 14:36:28.713376 6331 external_mini_cluster.cc:1428] Setting key b8559cdccbd56a85a9c2f91e80275b4f
I20260810 14:36:28.716487 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.46.195:0
--local_ip_for_outbound_sockets=127.6.46.195
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43161
--builtin_ntp_servers=127.6.46.212:43115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 14:36:28.721639 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:28.710828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.194:47771 (local address 127.6.46.254:43161)
0810 14:36:28.711064 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:28.711070 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:28.712492 (+ 1422us) server_negotiation.cc:407] Connection header received
0810 14:36:28.714795 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:28.714799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:28.714877 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:28.715018 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:28.716305 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:28.717051 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:28.719113 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:28.719512 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:28.719605 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:28.721010 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:28.721015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:28.721018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:28.721035 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:28.721065 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:28.721076 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:28.721141 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:28.721350 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 14:36:28.721421 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:36:28.722911 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:28.711388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:43161 (local address 127.6.46.194:47771)
0810 14:36:28.712320 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:28.712370 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:28.714497 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:28.715241 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:28.715252 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:28.715360 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:28.716123 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:28.716151 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:28.717193 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:28.717199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:28.718979 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:28.718994 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:28.719727 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:28.719733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:28.719999 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:28.720702 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:28.720705 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:28.720752 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:28.720839 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:28.721158 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:28.721169 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:28.721176 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 14:36:28.721338 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 14:36:28.721693 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":273,"threads_started":1}
I20260810 14:36:28.724138 7817 heartbeater.cc:347] Connected to a master server at 127.6.46.254:43161
I20260810 14:36:28.724426 7817 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:28.725270 7817 heartbeater.cc:511] Master 127.6.46.254:43161 requested a full tablet report, sending...
I20260810 14:36:28.726675 7268 ts_manager.cc:194] Registered new tserver with Master: 5dead38db0cf4b06a4a30c8f91a2478a (127.6.46.194:36619)
I20260810 14:36:28.727471 7268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.46.194:47771
W20260810 14:36:29.044960 7825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:29.045207 7825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:29.045326 7825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:29.052284 7825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:29.052374 7825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:29.052525 7825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.46.195
I20260810 14:36:29.061172 7825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.46.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.46.195
--webserver_port=0
--tserver_master_addrs=127.6.46.254:43161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.46.195
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:29.062379 7825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:29.064083 7825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:29.076333 7835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:29.076340 7838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:29.076370 7842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:29.077060 7825 server_base.cc:1034] running on GCE node
I20260810 14:36:29.077636 7825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:29.078614 7825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:29.079807 7825 hybrid_clock.cc:648] HybridClock initialized: now 1786372589079764 us; error 37 us; skew 500 ppm
I20260810 14:36:29.080085 7825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 14:36:29.082219 7825 webserver.cc:460] Webserver started at http://127.6.46.195:46877/ using document root <none> and password file <none>
I20260810 14:36:29.082885 7825 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:29.082984 7825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:29.083284 7825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:29.085978 7825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/data/instance:
uuid: "3dbaedfe079e48e8b5c237ea20828e63"
format_stamp: "Formatted at 2026-08-10 14:36:29 on dist-test-slave-q9h9"
server_key: "8c1df0cc4f8d41ceac82f91113e9b2e4"
server_key_iv: "48a318180cb651d605c7dda5c4c35d21"
server_key_version: "encryptionkey@0"
I20260810 14:36:29.086778 7825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance:
uuid: "3dbaedfe079e48e8b5c237ea20828e63"
format_stamp: "Formatted at 2026-08-10 14:36:29 on dist-test-slave-q9h9"
server_key: "8c1df0cc4f8d41ceac82f91113e9b2e4"
server_key_iv: "48a318180cb651d605c7dda5c4c35d21"
server_key_version: "encryptionkey@0"
I20260810 14:36:29.091495 7825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:36:29.094765 7851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:29.095952 7825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:29.096101 7825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/wal
uuid: "3dbaedfe079e48e8b5c237ea20828e63"
format_stamp: "Formatted at 2026-08-10 14:36:29 on dist-test-slave-q9h9"
server_key: "8c1df0cc4f8d41ceac82f91113e9b2e4"
server_key_iv: "48a318180cb651d605c7dda5c4c35d21"
server_key_version: "encryptionkey@0"
I20260810 14:36:29.096303 7825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:29.107963 7825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:29.108626 7825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:29.108863 7825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:29.109740 7825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 14:36:29.111132 7825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 14:36:29.111222 7825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:29.111299 7825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 14:36:29.111351 7825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:29.140443 7825 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.195:45489
I20260810 14:36:29.140503 8044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.195:45489 every 8 connection(s)
I20260810 14:36:29.141605 7825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/data/info.pb
I20260810 14:36:29.147189 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 7825
I20260810 14:36:29.147346 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/wal/instance
I20260810 14:36:29.147917 6331 external_mini_cluster.cc:1428] Setting key a637dae665a76be486a8d33b39c398ce
I20260810 14:36:29.153798 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.144073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:39979 (local address 127.6.46.254:43161)
0810 14:36:29.144267 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:29.144272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:29.145547 (+ 1275us) server_negotiation.cc:407] Connection header received
0810 14:36:29.147317 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:29.147323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:29.147404 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:29.147500 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:29.148930 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.149880 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.151531 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.151937 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.152030 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.153178 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:29.153182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:29.153185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:29.153202 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:29.153231 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:29.153243 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:29.153309 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:29.153479 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 14:36:29.153550 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:36:29.154740 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.144534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:43161 (local address 127.6.46.195:39979)
0810 14:36:29.145387 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:29.145429 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:29.147053 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:29.147762 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:29.147774 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:29.147891 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:36:29.148665 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:29.148693 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.150048 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.150054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:29.151402 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:29.151416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.152139 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.152145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:29.152349 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.152898 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 14:36:29.152901 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:36:29.152942 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 14:36:29.153018 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:36:29.153315 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:36:29.153322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:36:29.153327 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 14:36:29.153472 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 14:36:29.153733 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":273,"threads_started":1}
I20260810 14:36:29.155918 8048 heartbeater.cc:347] Connected to a master server at 127.6.46.254:43161
I20260810 14:36:29.156162 8048 heartbeater.cc:464] Registering TS with master...
I20260810 14:36:29.156781 8048 heartbeater.cc:511] Master 127.6.46.254:43161 requested a full tablet report, sending...
I20260810 14:36:29.158010 7268 ts_manager.cc:194] Registered new tserver with Master: 3dbaedfe079e48e8b5c237ea20828e63 (127.6.46.195:45489)
I20260810 14:36:29.158815 7268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.46.195:39979
I20260810 14:36:29.162704 6331 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 14:36:29.171679 7582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.166495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59128 (local address 127.6.46.254:43161)
0810 14:36:29.166657 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:29.166662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:29.167096 (+ 434us) server_negotiation.cc:407] Connection header received
0810 14:36:29.167217 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:29.167221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:29.167286 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:29.167362 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:29.167871 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.168538 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.169764 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.170171 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.170261 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.171133 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:29.171137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:29.171140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:29.171154 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:29.171178 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:29.171188 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:29.171246 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:29.171335 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 14:36:29.171395 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:36:29.174964 7268 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 14:36:29.181473 7268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 14:36:29.203663 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.193507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:36619 (local address 127.0.0.1:40078)
0810 14:36:29.194042 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:29.194064 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:29.194166 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:29.195241 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:29.195246 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:29.195273 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:29.195724 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:29.195736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.198081 (+ 2345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.198088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:29.200694 (+ 2606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:29.200708 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.202695 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.202702 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:29.202924 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.202977 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 14:36:29.203088 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 14:36:29.203259 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":24,"thread_start_us":251,"threads_started":1}
I20260810 14:36:29.204806 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.192247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.195:45489 (local address 127.0.0.1:49896)
0810 14:36:29.193171 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:29.193225 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:29.193382 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:29.194322 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:29.194329 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:29.194372 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:29.194765 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:29.194778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.198384 (+ 3606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.198391 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:29.200876 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:29.200893 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.203985 (+ 3092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.203992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:29.204197 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.204228 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 14:36:29.204288 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 14:36:29.204418 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":4,"thread_start_us":501,"threads_started":1}
I20260810 14:36:29.205401 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.193985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.6.46.194:36619)
0810 14:36:29.194642 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:29.194650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:29.194677 (+ 27us) server_negotiation.cc:407] Connection header received
0810 14:36:29.194794 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:29.194803 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:29.195029 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:29.195961 (+ 932us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:29.196250 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.197877 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.200958 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.202469 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.204262 (+ 1793us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.204557 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:29.204633 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 14:36:29.204799 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":528,"threads_started":1}
I20260810 14:36:29.206872 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.192426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49896 (local address 127.6.46.195:45489)
0810 14:36:29.193655 (+ 1229us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:29.193664 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:29.193703 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:36:29.193797 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:29.193805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:29.194069 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:29.194981 (+ 912us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:29.196444 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.198183 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.202283 (+ 4100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.203744 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.205032 (+ 1288us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.206340 (+ 1308us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:29.206409 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 14:36:29.206563 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1046,"thread_start_us":897,"threads_started":1}
I20260810 14:36:29.210402 7928 tablet_service.cc:1467] Processing CreateTablet for tablet 5ead1324205b4d669247c68fc81bda03 (DEFAULT_TABLE table=test-table [id=e77ca5793c9f408ea1dbc4720aa090dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:29.210219 7697 tablet_service.cc:1467] Processing CreateTablet for tablet 5ead1324205b4d669247c68fc81bda03 (DEFAULT_TABLE table=test-table [id=e77ca5793c9f408ea1dbc4720aa090dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:29.211771 7928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ead1324205b4d669247c68fc81bda03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:29.211771 7697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ead1324205b4d669247c68fc81bda03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:29.211894 8069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.194291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:41955 (local address 127.0.0.1:45790)
0810 14:36:29.203872 (+ 9581us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:29.203888 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:29.203984 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:29.206657 (+ 2673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:29.206661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:29.206683 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:29.206825 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:29.206836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.208330 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.208335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:29.209497 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:29.209517 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.211207 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.211213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:29.211391 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.211422 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 14:36:29.211510 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 14:36:29.211633 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9491,"thread_start_us":11919,"threads_started":1}
I20260810 14:36:29.211984 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.194402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45790 (local address 127.6.46.193:41955)
0810 14:36:29.206158 (+ 11756us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:29.206167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:29.206196 (+ 29us) server_negotiation.cc:407] Connection header received
0810 14:36:29.206288 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:29.206295 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:29.206490 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:29.206652 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:29.206955 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.208205 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.209643 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.211010 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.211179 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.211391 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:29.211553 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 14:36:29.211695 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11577,"thread_start_us":11491,"threads_started":1}
I20260810 14:36:29.214674 7470 tablet_service.cc:1467] Processing CreateTablet for tablet 5ead1324205b4d669247c68fc81bda03 (DEFAULT_TABLE table=test-table [id=e77ca5793c9f408ea1dbc4720aa090dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 14:36:29.215705 7470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ead1324205b4d669247c68fc81bda03. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:29.225889 8076 tablet_bootstrap.cc:492] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a: Bootstrap starting.
I20260810 14:36:29.226824 8077 tablet_bootstrap.cc:492] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63: Bootstrap starting.
I20260810 14:36:29.228911 8078 tablet_bootstrap.cc:492] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a: Bootstrap starting.
I20260810 14:36:29.229712 8076 tablet_bootstrap.cc:654] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:29.230309 8077 tablet_bootstrap.cc:654] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:29.230845 8076 log.cc:824] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:29.231509 8077 log.cc:824] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:29.232270 8078 tablet_bootstrap.cc:654] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:29.233511 8078 log.cc:824] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:29.234040 8076 tablet_bootstrap.cc:492] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a: No bootstrap required, opened a new log
I20260810 14:36:29.234114 8077 tablet_bootstrap.cc:492] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63: No bootstrap required, opened a new log
I20260810 14:36:29.234344 8076 ts_tablet_manager.cc:1397] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260810 14:36:29.234424 8077 ts_tablet_manager.cc:1397] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260810 14:36:29.236342 8078 tablet_bootstrap.cc:492] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a: No bootstrap required, opened a new log
I20260810 14:36:29.236585 8078 ts_tablet_manager.cc:1397] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260810 14:36:29.240926 8077 raft_consensus.cc:348] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaedfe079e48e8b5c237ea20828e63" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 45489 } } peers { permanent_uuid: "5dead38db0cf4b06a4a30c8f91a2478a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36619 } } peers { permanent_uuid: "bf2a475d0b4c42b6b67075b57ccbc46a" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 41955 } }
I20260810 14:36:29.241254 8077 raft_consensus.cc:374] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:29.241384 8077 raft_consensus.cc:729] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dbaedfe079e48e8b5c237ea20828e63, State: Initialized, Role: FOLLOWER
I20260810 14:36:29.241910 8077 consensus_queue.cc:260] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaedfe079e48e8b5c237ea20828e63" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 45489 } } peers { permanent_uuid: "5dead38db0cf4b06a4a30c8f91a2478a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36619 } } peers { permanent_uuid: "bf2a475d0b4c42b6b67075b57ccbc46a" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 41955 } }
I20260810 14:36:29.243757 8048 heartbeater.cc:503] Master 127.6.46.254:43161 was elected leader, sending a full tablet report...
I20260810 14:36:29.244248 8077 ts_tablet_manager.cc:1428] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20260810 14:36:29.244886 8076 raft_consensus.cc:348] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaedfe079e48e8b5c237ea20828e63" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 45489 } } peers { permanent_uuid: "5dead38db0cf4b06a4a30c8f91a2478a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36619 } } peers { permanent_uuid: "bf2a475d0b4c42b6b67075b57ccbc46a" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 41955 } }
I20260810 14:36:29.245296 8076 raft_consensus.cc:374] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:29.245398 8076 raft_consensus.cc:729] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dead38db0cf4b06a4a30c8f91a2478a, State: Initialized, Role: FOLLOWER
I20260810 14:36:29.245956 8076 consensus_queue.cc:260] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaedfe079e48e8b5c237ea20828e63" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 45489 } } peers { permanent_uuid: "5dead38db0cf4b06a4a30c8f91a2478a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36619 } } peers { permanent_uuid: "bf2a475d0b4c42b6b67075b57ccbc46a" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 41955 } }
I20260810 14:36:29.246312 8078 raft_consensus.cc:348] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaedfe079e48e8b5c237ea20828e63" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 45489 } } peers { permanent_uuid: "5dead38db0cf4b06a4a30c8f91a2478a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36619 } } peers { permanent_uuid: "bf2a475d0b4c42b6b67075b57ccbc46a" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 41955 } }
I20260810 14:36:29.246788 8078 raft_consensus.cc:374] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:29.246934 8078 raft_consensus.cc:729] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf2a475d0b4c42b6b67075b57ccbc46a, State: Initialized, Role: FOLLOWER
I20260810 14:36:29.247376 7817 heartbeater.cc:503] Master 127.6.46.254:43161 was elected leader, sending a full tablet report...
I20260810 14:36:29.247496 8078 consensus_queue.cc:260] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaedfe079e48e8b5c237ea20828e63" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 45489 } } peers { permanent_uuid: "5dead38db0cf4b06a4a30c8f91a2478a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36619 } } peers { permanent_uuid: "bf2a475d0b4c42b6b67075b57ccbc46a" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 41955 } }
I20260810 14:36:29.247726 8076 ts_tablet_manager.cc:1428] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260810 14:36:29.248559 7580 heartbeater.cc:503] Master 127.6.46.254:43161 was elected leader, sending a full tablet report...
I20260810 14:36:29.249774 8078 ts_tablet_manager.cc:1428] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
W20260810 14:36:29.262638 7581 tablet.cc:2369] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 14:36:29.393364 8049 tablet.cc:2369] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:36:29.458573 8087 raft_consensus.cc:482] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 14:36:29.458882 8087 raft_consensus.cc:504] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaedfe079e48e8b5c237ea20828e63" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 45489 } } peers { permanent_uuid: "5dead38db0cf4b06a4a30c8f91a2478a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36619 } } peers { permanent_uuid: "bf2a475d0b4c42b6b67075b57ccbc46a" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 41955 } }
W20260810 14:36:29.460254 7818 tablet.cc:2369] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:36:29.460577 8087 leader_election.cc:288] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5dead38db0cf4b06a4a30c8f91a2478a (127.6.46.194:36619), bf2a475d0b4c42b6b67075b57ccbc46a (127.6.46.193:41955)
I20260810 14:36:29.467170 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.461059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:51363 (local address 127.6.46.194:36619)
0810 14:36:29.461253 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:29.461259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:29.461279 (+ 20us) server_negotiation.cc:407] Connection header received
0810 14:36:29.461417 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:29.461422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:29.461503 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:29.461596 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:29.462174 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.463182 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.465221 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.466398 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.466524 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.466572 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:29.466893 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 14:36:29.466977 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 14:36:29.467298 8050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.460938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.194:36619 (local address 127.6.46.195:51363)
0810 14:36:29.461135 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:29.461156 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:29.461279 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:29.461681 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:29.461688 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:29.461724 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:29.461977 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:29.461989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.463308 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.463313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:29.465044 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:29.465058 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.466533 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.466539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:29.466736 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.466772 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 14:36:29.466874 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 14:36:29.467013 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 14:36:29.468133 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.461662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.46.195:59529 (local address 127.6.46.193:41955)
0810 14:36:29.461860 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:29.461866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:29.462307 (+ 441us) server_negotiation.cc:407] Connection header received
0810 14:36:29.462452 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:29.462455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:29.462528 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:29.462597 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 14:36:29.463186 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.464185 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.466286 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.467355 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.467504 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.467563 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:29.467832 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 14:36:29.467955 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:36:29.468135 8090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.461347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.193:41955 (local address 127.6.46.195:59529)
0810 14:36:29.462199 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0810 14:36:29.462216 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:36:29.462344 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:36:29.462631 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:36:29.462635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:36:29.462671 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 14:36:29.463041 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:29.463052 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.464335 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.464339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:29.466095 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:36:29.466111 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.467533 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.467539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:36:29.467695 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.467718 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 14:36:29.467785 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 14:36:29.467865 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"thread_start_us":528,"threads_started":1}
I20260810 14:36:29.468410 7731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ead1324205b4d669247c68fc81bda03" candidate_uuid: "3dbaedfe079e48e8b5c237ea20828e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dead38db0cf4b06a4a30c8f91a2478a" is_pre_election: true
I20260810 14:36:29.468899 7731 raft_consensus.cc:2454] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dbaedfe079e48e8b5c237ea20828e63 in term 0.
I20260810 14:36:29.469302 7509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ead1324205b4d669247c68fc81bda03" candidate_uuid: "3dbaedfe079e48e8b5c237ea20828e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2a475d0b4c42b6b67075b57ccbc46a" is_pre_election: true
I20260810 14:36:29.469609 7858 leader_election.cc:302] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dbaedfe079e48e8b5c237ea20828e63, 5dead38db0cf4b06a4a30c8f91a2478a; no voters:
I20260810 14:36:29.469858 7509 raft_consensus.cc:2454] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dbaedfe079e48e8b5c237ea20828e63 in term 0.
I20260810 14:36:29.470036 8087 raft_consensus.cc:2781] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 14:36:29.470172 8087 raft_consensus.cc:482] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 14:36:29.470280 8087 raft_consensus.cc:3037] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:29.472591 8087 raft_consensus.cc:504] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaedfe079e48e8b5c237ea20828e63" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 45489 } } peers { permanent_uuid: "5dead38db0cf4b06a4a30c8f91a2478a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36619 } } peers { permanent_uuid: "bf2a475d0b4c42b6b67075b57ccbc46a" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 41955 } }
I20260810 14:36:29.473109 8087 leader_election.cc:288] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [CANDIDATE]: Term 1 election: Requested vote from peers 5dead38db0cf4b06a4a30c8f91a2478a (127.6.46.194:36619), bf2a475d0b4c42b6b67075b57ccbc46a (127.6.46.193:41955)
I20260810 14:36:29.473489 7731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ead1324205b4d669247c68fc81bda03" candidate_uuid: "3dbaedfe079e48e8b5c237ea20828e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5dead38db0cf4b06a4a30c8f91a2478a"
I20260810 14:36:29.473541 7509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ead1324205b4d669247c68fc81bda03" candidate_uuid: "3dbaedfe079e48e8b5c237ea20828e63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf2a475d0b4c42b6b67075b57ccbc46a"
I20260810 14:36:29.473649 7731 raft_consensus.cc:3037] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:29.473721 7509 raft_consensus.cc:3037] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:29.475813 7731 raft_consensus.cc:2454] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dbaedfe079e48e8b5c237ea20828e63 in term 1.
I20260810 14:36:29.475978 7509 raft_consensus.cc:2454] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dbaedfe079e48e8b5c237ea20828e63 in term 1.
I20260810 14:36:29.476183 7858 leader_election.cc:302] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dbaedfe079e48e8b5c237ea20828e63, 5dead38db0cf4b06a4a30c8f91a2478a; no voters:
I20260810 14:36:29.476439 8087 raft_consensus.cc:2781] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:29.476900 8087 raft_consensus.cc:686] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [term 1 LEADER]: Becoming Leader. State: Replica: 3dbaedfe079e48e8b5c237ea20828e63, State: Running, Role: LEADER
I20260810 14:36:29.477396 8087 consensus_queue.cc:237] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaedfe079e48e8b5c237ea20828e63" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 45489 } } peers { permanent_uuid: "5dead38db0cf4b06a4a30c8f91a2478a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36619 } } peers { permanent_uuid: "bf2a475d0b4c42b6b67075b57ccbc46a" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 41955 } }
I20260810 14:36:29.481721 7267 catalog_manager.cc:5495] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dbaedfe079e48e8b5c237ea20828e63 (127.6.46.195). New cstate: current_term: 1 leader_uuid: "3dbaedfe079e48e8b5c237ea20828e63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dbaedfe079e48e8b5c237ea20828e63" member_type: VOTER last_known_addr { host: "127.6.46.195" port: 45489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5dead38db0cf4b06a4a30c8f91a2478a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf2a475d0b4c42b6b67075b57ccbc46a" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 41955 } health_report { overall_health: UNKNOWN } } }
I20260810 14:36:29.564525 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.559633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49904 (local address 127.6.46.195:45489)
0810 14:36:29.559868 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:29.559873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:29.559891 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:36:29.560072 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:29.560077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:29.560157 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:29.560275 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:36:29.560542 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.561249 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.562583 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.563001 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.563106 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.563498 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:36:29.564034 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:36:29.564148 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:29.564218 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 14:36:29.564281 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 14:36:29.572242 7509 raft_consensus.cc:1261] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a [term 1 FOLLOWER]: Refusing update from remote peer 3dbaedfe079e48e8b5c237ea20828e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:29.572288 7731 raft_consensus.cc:1261] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a [term 1 FOLLOWER]: Refusing update from remote peer 3dbaedfe079e48e8b5c237ea20828e63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:36:29.573107 8096 consensus_queue.cc:1035] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf2a475d0b4c42b6b67075b57ccbc46a" member_type: VOTER last_known_addr { host: "127.6.46.193" port: 41955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:29.573460 8087 consensus_queue.cc:1035] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dead38db0cf4b06a4a30c8f91a2478a" member_type: VOTER last_known_addr { host: "127.6.46.194" port: 36619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:36:29.593433 8104 mvcc.cc:205] Tried to move back new op lower bound from 7316982126878273536 to 7316982126503489536. Current Snapshot: MvccSnapshot[applied={T|T < 7316982126878273536}]
I20260810 14:36:29.594684 7267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:29.594960 7267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 14:36:29.599287 7267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 787867886369437596d5f1bc6def73e9: Sending DeleteTablet for 3 replicas of tablet 5ead1324205b4d669247c68fc81bda03
I20260810 14:36:29.600523 7470 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ead1324205b4d669247c68fc81bda03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:29 UTC) from {username='slave'} at 127.0.0.1:45790
I20260810 14:36:29.600534 7697 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ead1324205b4d669247c68fc81bda03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:29 UTC) from {username='slave'} at 127.0.0.1:40078
I20260810 14:36:29.600706 7695 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ead1324205b4d669247c68fc81bda03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:29 UTC) from {username='slave'} at 127.0.0.1:40078
I20260810 14:36:29.601375 7928 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ead1324205b4d669247c68fc81bda03 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 14:36:29 UTC) from {username='slave'} at 127.0.0.1:49896
I20260810 14:36:29.601459 8118 tablet_replica.cc:330] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a: stopping tablet replica
I20260810 14:36:29.601847 8118 raft_consensus.cc:2229] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:29.602358 8118 raft_consensus.cc:2258] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:29.602675 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 7354
I20260810 14:36:29.611109 8121 tablet_replica.cc:330] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63: stopping tablet replica
I20260810 14:36:29.611121 8118 ts_tablet_manager.cc:1905] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:29.611569 8121 raft_consensus.cc:2229] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [term 1 LEADER]: Raft consensus shutting down.
I20260810 14:36:29.612169 8121 raft_consensus.cc:2258] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:29.613895 8121 ts_tablet_manager.cc:1905] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:29.614208 8120 tablet_replica.cc:330] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a: stopping tablet replica
I20260810 14:36:29.616397 8120 raft_consensus.cc:2229] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 14:36:29.616895 8120 raft_consensus.cc:2258] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 14:36:29.617430 8118 ts_tablet_manager.cc:1918] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:29.617739 8118 log.cc:1196] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-0/wal/wals/5ead1324205b4d669247c68fc81bda03
I20260810 14:36:29.618342 8118 ts_tablet_manager.cc:1939] T 5ead1324205b4d669247c68fc81bda03 P bf2a475d0b4c42b6b67075b57ccbc46a: Deleting consensus metadata
I20260810 14:36:29.619676 8120 ts_tablet_manager.cc:1905] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 14:36:29.620467 7248 catalog_manager.cc:4841] TS bf2a475d0b4c42b6b67075b57ccbc46a (127.6.46.193:41955): tablet 5ead1324205b4d669247c68fc81bda03 (table test-table [id=e77ca5793c9f408ea1dbc4720aa090dc]) successfully deleted
I20260810 14:36:29.620803 8121 ts_tablet_manager.cc:1918] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:29.620960 8121 log.cc:1196] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-2/wal/wals/5ead1324205b4d669247c68fc81bda03
I20260810 14:36:29.621408 8121 ts_tablet_manager.cc:1939] T 5ead1324205b4d669247c68fc81bda03 P 3dbaedfe079e48e8b5c237ea20828e63: Deleting consensus metadata
I20260810 14:36:29.622900 7248 catalog_manager.cc:4841] TS 3dbaedfe079e48e8b5c237ea20828e63 (127.6.46.195:45489): tablet 5ead1324205b4d669247c68fc81bda03 (table test-table [id=e77ca5793c9f408ea1dbc4720aa090dc]) successfully deleted
I20260810 14:36:29.623955 8075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.604924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.6.46.193:41955)
0810 14:36:29.605100 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:29.605106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:29.610541 (+ 5435us) server_negotiation.cc:407] Connection header received
0810 14:36:29.610815 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:29.610819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:29.610914 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:29.610987 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:29.612541 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.613685 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.617319 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.617976 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.618111 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.622462 (+ 4351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:29.622472 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:29.622486 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:29.623154 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:29.623210 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:29.623235 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:29.623311 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:29.623525 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 14:36:29.623630 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:36:29.624531 7574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:29.626363 8120 ts_tablet_manager.cc:1918] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 14:36:29.626533 8120 log.cc:1196] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a: Deleting WAL directory at /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786372275129649-6331-0/minicluster-data/ts-1/wal/wals/5ead1324205b4d669247c68fc81bda03
I20260810 14:36:29.627058 8120 ts_tablet_manager.cc:1939] T 5ead1324205b4d669247c68fc81bda03 P 5dead38db0cf4b06a4a30c8f91a2478a: Deleting consensus metadata
I20260810 14:36:29.628757 7248 catalog_manager.cc:4841] TS 5dead38db0cf4b06a4a30c8f91a2478a (127.6.46.194:36619): tablet 5ead1324205b4d669247c68fc81bda03 (table test-table [id=e77ca5793c9f408ea1dbc4720aa090dc]) successfully deleted
W20260810 14:36:29.629407 7248 catalog_manager.cc:4808] TS 5dead38db0cf4b06a4a30c8f91a2478a (127.6.46.194:36619): delete failed for tablet 5ead1324205b4d669247c68fc81bda03 because the tablet was not found. No further retry: Not found: Tablet not found: 5ead1324205b4d669247c68fc81bda03
I20260810 14:36:29.720139 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 7354
I20260810 14:36:29.745893 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 7586
I20260810 14:36:29.753027 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.746890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40084 (local address 127.6.46.194:36619)
0810 14:36:29.747079 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:29.747085 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:29.747103 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:36:29.747153 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:29.747157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:29.747246 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:29.747336 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:29.747762 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.748640 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.749930 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.750524 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.750630 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.751664 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:29.751673 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:29.751688 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:29.752267 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:29.752315 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:29.752335 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:29.752433 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:29.752582 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 14:36:29.752694 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 14:36:29.753819 7803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:29.840946 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 7586
I20260810 14:36:29.867524 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 7825
I20260810 14:36:29.875164 8067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:29.868443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.6.46.195:45489)
0810 14:36:29.868604 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:29.868608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:29.868627 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:36:29.868690 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:29.868694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:29.868789 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:29.868877 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:29.869368 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.870200 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.872084 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:29.872537 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:29.872630 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:29.873767 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:29.873775 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:29.873789 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:29.874413 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:29.874460 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:29.874484 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:29.874569 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:29.874749 (+ 180us) server_negotiation.cc:299] Negotiation successful
0810 14:36:29.874830 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:36:29.875914 8039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:29.990350 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 7825
I20260810 14:36:30.013056 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 7221
I20260810 14:36:30.014168 7313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:30.119027 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 7221
2026-08-10T14:36:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2885 ms)
[----------] 6 tests from EncryptionPolicyTest (22292 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:30 dist-test-slave-q9h9 krb5kdc[8138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:30 dist-test-slave-q9h9 krb5kdc[8138](info): set up 2 sockets
Aug 10 14:36:30 dist-test-slave-q9h9 krb5kdc[8138](info): commencing operation
krb5kdc: starting...
W20260810 14:36:31.359172 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.190s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:31 dist-test-slave-q9h9 krb5kdc[8138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:36:31.619609 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:31.945410 8159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:31.945689 8159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:31.945812 8159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:31.952956 8159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:31.953063 8159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:31.953147 8159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:31.953223 8159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:31.953281 8159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:31.963104 8159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:31.964380 8159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:31.965703 8159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:31.977048 8168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:31.977093 8173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:31.977057 8169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:31.977851 8159 server_base.cc:1034] running on GCE node
I20260810 14:36:31.978642 8159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:31.979921 8159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:31.981143 8159 hybrid_clock.cc:648] HybridClock initialized: now 1786372591981102 us; error 30 us; skew 500 ppm
I20260810 14:36:31.981482 8159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:31 dist-test-slave-q9h9 krb5kdc[8138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:31.984524 8159 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:36:31.985916 8159 webserver.cc:460] Webserver started at http://127.0.0.1:35777/ using document root <none> and password file <none>
I20260810 14:36:31.986598 8159 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:31.986724 8159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:31.987015 8159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:31.989905 8159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "2eac689eb3654d3fbd3e95bf7e742669"
format_stamp: "Formatted at 2026-08-10 14:36:31 on dist-test-slave-q9h9"
server_key: "45d14d63e6a9b0ac1fcafb30c5521577"
server_key_iv: "cf89f57c61203e37c0738536093e1ad6"
server_key_version: "encryptionkey@0"
I20260810 14:36:31.990694 8159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "2eac689eb3654d3fbd3e95bf7e742669"
format_stamp: "Formatted at 2026-08-10 14:36:31 on dist-test-slave-q9h9"
server_key: "45d14d63e6a9b0ac1fcafb30c5521577"
server_key_iv: "cf89f57c61203e37c0738536093e1ad6"
server_key_version: "encryptionkey@0"
I20260810 14:36:31.995568 8159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 14:36:31.998876 8186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:32.000074 8159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:32.000239 8159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "2eac689eb3654d3fbd3e95bf7e742669"
format_stamp: "Formatted at 2026-08-10 14:36:31 on dist-test-slave-q9h9"
server_key: "45d14d63e6a9b0ac1fcafb30c5521577"
server_key_iv: "cf89f57c61203e37c0738536093e1ad6"
server_key_version: "encryptionkey@0"
I20260810 14:36:32.000411 8159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:32.030732 8159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:32.034969 8159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:32.035264 8159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:32.052572 8159 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35477
I20260810 14:36:32.052611 8278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35477 every 8 connection(s)
I20260810 14:36:32.053880 8159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:32.055088 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 8159
I20260810 14:36:32.055321 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:32.056156 6331 external_mini_cluster.cc:1428] Setting key 6ffb6749cc839a8635e0d11aef783f5d
I20260810 14:36:32.058531 8279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 14:36:32 dist-test-slave-q9h9 krb5kdc[8138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 14:36:32.071188 8279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669: Bootstrap starting.
I20260810 14:36:32.074275 8279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:32.075340 8279 log.cc:824] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:32.078089 8279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669: No bootstrap required, opened a new log
I20260810 14:36:32.078658 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:32.058193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34783 (local address 127.0.0.1:35477)
0810 14:36:32.058980 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:32.058991 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:32.059030 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:36:32.060632 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:32.060653 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:32.061020 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:32.061326 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:32.063151 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:32.064469 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:32.066218 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:32.066854 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:32.066980 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:32.071516 (+ 4536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:32.071524 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:32.071537 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:32.071584 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:32.074622 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:32.075418 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:32.075424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:32.075430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:32.075616 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:32.076054 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:32.076057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:32.076059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:32.076554 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:32.076855 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:32.077331 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 14:36:32.077633 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":361,"threads_started":1}
I20260810 14:36:32.082257 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:32.085727 8279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eac689eb3654d3fbd3e95bf7e742669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } }
I20260810 14:36:32.086097 8279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:32.086202 8279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eac689eb3654d3fbd3e95bf7e742669, State: Initialized, Role: FOLLOWER
I20260810 14:36:32.086858 8279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eac689eb3654d3fbd3e95bf7e742669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } }
I20260810 14:36:32.087162 8279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:32.087309 8279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:32.087472 8279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:32.090282 8279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eac689eb3654d3fbd3e95bf7e742669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } }
I20260810 14:36:32.090839 8279 leader_election.cc:302] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2eac689eb3654d3fbd3e95bf7e742669; no voters:
I20260810 14:36:32.091459 8279 leader_election.cc:288] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:32.091583 8297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:32.092157 8297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [term 1 LEADER]: Becoming Leader. State: Replica: 2eac689eb3654d3fbd3e95bf7e742669, State: Running, Role: LEADER
I20260810 14:36:32.092756 8297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eac689eb3654d3fbd3e95bf7e742669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } }
I20260810 14:36:32.096724 8297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eac689eb3654d3fbd3e95bf7e742669. Latest consensus state: current_term: 1 leader_uuid: "2eac689eb3654d3fbd3e95bf7e742669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eac689eb3654d3fbd3e95bf7e742669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } } }
I20260810 14:36:32.098312 8297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:32.097950 8302 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2eac689eb3654d3fbd3e95bf7e742669" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eac689eb3654d3fbd3e95bf7e742669" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35477 } } }
I20260810 14:36:32.098706 8302 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:32.101939 8279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:32.102182 8305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:32.103538 8284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:32.086756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54931 (local address 127.0.0.1:35477)
0810 14:36:32.086902 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:32.086908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:32.087334 (+ 426us) server_negotiation.cc:407] Connection header received
0810 14:36:32.087498 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:32.087503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:32.087567 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:32.087655 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:32.089478 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:32.090294 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:32.092090 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:32.092870 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:32.092968 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:32.096364 (+ 3396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:32.096371 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:32.096377 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:32.096416 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:32.100091 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:32.100950 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:32.100954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:32.100957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:32.101104 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:32.101593 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:32.101597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:32.101601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:32.101949 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:32.102282 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:32.102890 (+ 608us) server_negotiation.cc:299] Negotiation successful
0810 14:36:32.103106 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:36:32.108026 8305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:32.117082 8305 catalog_manager.cc:1347] Generated new cluster ID: 81541b51a9fd4e71b72c868d84358d69
I20260810 14:36:32.117192 8305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:32.139602 8305 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:32.140868 8305 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:32.148144 8305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2eac689eb3654d3fbd3e95bf7e742669: Generated new TSK 0
I20260810 14:36:32.149113 8305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:32.163184 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8159
I20260810 14:36:32.164471 8267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:32.254050 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 8159
W20260810 14:36:32.278507 8132 connection.cc:535] client connection to 127.0.0.1:35477 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35477): Connection reset by peer (error 104)
2026-08-10T14:36:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2128 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:32 dist-test-slave-q9h9 krb5kdc[8335](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:32 dist-test-slave-q9h9 krb5kdc[8335](info): set up 2 sockets
Aug 10 14:36:32 dist-test-slave-q9h9 krb5kdc[8335](info): commencing operation
krb5kdc: starting...
W20260810 14:36:33.498631 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.008s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:33 dist-test-slave-q9h9 krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:36:33.785099 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:34.111368 8354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:34.111810 8354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:34.111974 8354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:34.119094 8354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:34.119217 8354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:34.119300 8354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:34.119385 8354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:34.119450 8354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:34.129226 8354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:34.130514 8354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:34.131819 8354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:34.143222 8371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:34.143184 8364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:34.143246 8366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:34.143929 8354 server_base.cc:1034] running on GCE node
I20260810 14:36:34.144683 8354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:34.146064 8354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:34.147260 8354 hybrid_clock.cc:648] HybridClock initialized: now 1786372594147207 us; error 47 us; skew 500 ppm
I20260810 14:36:34.147585 8354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:34 dist-test-slave-q9h9 krb5kdc[8335](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:34.154677 8354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:36:34.156059 8354 webserver.cc:460] Webserver started at http://127.0.0.1:46811/ using document root <none> and password file <none>
I20260810 14:36:34.156746 8354 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:34.156848 8354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:34.157094 8354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:34.159951 8354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "987bea8008654a65aa71cdfe78b10e5c"
format_stamp: "Formatted at 2026-08-10 14:36:34 on dist-test-slave-q9h9"
server_key: "3efa3415c2d53ec011ccd56bb8075d62"
server_key_iv: "6627b6c7b0978eff4c9e888fbf6db99c"
server_key_version: "encryptionkey@0"
I20260810 14:36:34.160650 8354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "987bea8008654a65aa71cdfe78b10e5c"
format_stamp: "Formatted at 2026-08-10 14:36:34 on dist-test-slave-q9h9"
server_key: "3efa3415c2d53ec011ccd56bb8075d62"
server_key_iv: "6627b6c7b0978eff4c9e888fbf6db99c"
server_key_version: "encryptionkey@0"
I20260810 14:36:34.165542 8354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 14:36:34.168874 8382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:34.170145 8354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:34.170298 8354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "987bea8008654a65aa71cdfe78b10e5c"
format_stamp: "Formatted at 2026-08-10 14:36:34 on dist-test-slave-q9h9"
server_key: "3efa3415c2d53ec011ccd56bb8075d62"
server_key_iv: "6627b6c7b0978eff4c9e888fbf6db99c"
server_key_version: "encryptionkey@0"
I20260810 14:36:34.170461 8354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:34.188391 8354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:34.192072 8354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:34.192312 8354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:34.209249 8354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43531
I20260810 14:36:34.209285 8465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43531 every 8 connection(s)
I20260810 14:36:34.210533 8354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:34.212059 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 8354
I20260810 14:36:34.212235 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:34.212677 6331 external_mini_cluster.cc:1428] Setting key 14d01e3fe8ff14ea3be6ff41922d7748
I20260810 14:36:34.214496 8469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:34.226936 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:34.214392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33447 (local address 127.0.0.1:43531)
0810 14:36:34.215179 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:34.215193 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:34.215239 (+ 46us) server_negotiation.cc:407] Connection header received
0810 14:36:34.217295 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:34.217321 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:34.217823 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:34.218259 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:34.218753 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:34.220745 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:34.222088 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:34.223080 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:34.223252 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:34.224337 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:34.224347 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:34.224367 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:34.225059 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:34.225152 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:34.225195 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:34.225357 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:34.225552 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 14:36:34.225722 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":329,"threads_started":1}
I20260810 14:36:34.228186 8469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c: Bootstrap starting.
I20260810 14:36:34.230741 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:34.231312 8469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:34.232507 8469 log.cc:824] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:34.235227 8469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c: No bootstrap required, opened a new log
I20260810 14:36:34.242004 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:34.235665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58371 (local address 127.0.0.1:43531)
0810 14:36:34.235845 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:34.235851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:34.236080 (+ 229us) server_negotiation.cc:407] Connection header received
0810 14:36:34.236228 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:34.236233 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:34.236306 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:34.236423 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:34.236853 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:34.238098 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:34.239418 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:34.240108 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:34.240255 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:34.241217 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:34.241228 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:34.241234 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:34.241255 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:34.241305 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:34.241327 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:34.241464 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:34.241574 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 14:36:34.241631 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 14:36:34.244745 8469 raft_consensus.cc:348] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987bea8008654a65aa71cdfe78b10e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43531 } }
I20260810 14:36:34.245090 8469 raft_consensus.cc:374] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:34.245172 8469 raft_consensus.cc:729] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 987bea8008654a65aa71cdfe78b10e5c, State: Initialized, Role: FOLLOWER
I20260810 14:36:34.245678 8469 consensus_queue.cc:260] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987bea8008654a65aa71cdfe78b10e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43531 } }
I20260810 14:36:34.245893 8469 raft_consensus.cc:388] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:34.245996 8469 raft_consensus.cc:482] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:34.246186 8469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:34.248333 8469 raft_consensus.cc:504] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987bea8008654a65aa71cdfe78b10e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43531 } }
I20260810 14:36:34.248790 8469 leader_election.cc:302] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 987bea8008654a65aa71cdfe78b10e5c; no voters:
I20260810 14:36:34.249409 8469 leader_election.cc:288] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:34.249491 8488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:34.250308 8488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [term 1 LEADER]: Becoming Leader. State: Replica: 987bea8008654a65aa71cdfe78b10e5c, State: Running, Role: LEADER
I20260810 14:36:34.250867 8488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987bea8008654a65aa71cdfe78b10e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43531 } }
I20260810 14:36:34.251528 8469 sys_catalog.cc:564] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:34.253350 8489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "987bea8008654a65aa71cdfe78b10e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987bea8008654a65aa71cdfe78b10e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43531 } } }
I20260810 14:36:34.253876 8489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:34.253405 8490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 987bea8008654a65aa71cdfe78b10e5c. Latest consensus state: current_term: 1 leader_uuid: "987bea8008654a65aa71cdfe78b10e5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "987bea8008654a65aa71cdfe78b10e5c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43531 } } }
I20260810 14:36:34.254163 8490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:34.255381 8497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:34.260581 8497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:34.268909 8497 catalog_manager.cc:1347] Generated new cluster ID: d4c258443436472a8477f9489a282fef
I20260810 14:36:34.269009 8497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:34.284821 8497 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:34.286527 8497 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:34.299527 8497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 987bea8008654a65aa71cdfe78b10e5c: Generated new TSK 0
I20260810 14:36:34.300276 8497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:34.312952 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8354
W20260810 14:36:34.314078 8458 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33447
W20260810 14:36:34.314733 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8354: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:36:34.314927 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 8354
W20260810 14:36:34.345382 8332 connection.cc:535] client connection to 127.0.0.1:43531 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43531): Connection reset by peer (error 104)
2026-08-10T14:36:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2066 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:34 dist-test-slave-q9h9 krb5kdc[8529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:34 dist-test-slave-q9h9 krb5kdc[8529](info): set up 2 sockets
Aug 10 14:36:34 dist-test-slave-q9h9 krb5kdc[8529](info): commencing operation
krb5kdc: starting...
W20260810 14:36:35.566642 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.203s user 0.006s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:35 dist-test-slave-q9h9 krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372595, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:36:35.830520 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:36.148902 8546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:36.149188 8546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:36.149331 8546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:36.156366 8546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:36.156491 8546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:36.156567 8546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:36.156648 8546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:36.156708 8546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:36.166309 8546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37255
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:36.167474 8546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:36.168738 8546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:36.179975 8556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:36.179973 8557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:36.180011 8560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:36.180863 8546 server_base.cc:1034] running on GCE node
I20260810 14:36:36.181569 8546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:36.182832 8546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:36.184037 8546 hybrid_clock.cc:648] HybridClock initialized: now 1786372596184001 us; error 28 us; skew 500 ppm
I20260810 14:36:36.184341 8546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:36 dist-test-slave-q9h9 krb5kdc[8529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:36.191049 8546 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:36:36.192394 8546 webserver.cc:460] Webserver started at http://127.0.0.1:43115/ using document root <none> and password file <none>
I20260810 14:36:36.193009 8546 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:36.193128 8546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:36.193397 8546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:36.196236 8546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "26ea528195824c24a58856832c8ae88c"
format_stamp: "Formatted at 2026-08-10 14:36:36 on dist-test-slave-q9h9"
server_key: "c25a8d75c439b4f45fc0b542fbdb75ff"
server_key_iv: "69e54f282dd67dfe941f276ee04153d9"
server_key_version: "encryptionkey@0"
I20260810 14:36:36.196946 8546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "26ea528195824c24a58856832c8ae88c"
format_stamp: "Formatted at 2026-08-10 14:36:36 on dist-test-slave-q9h9"
server_key: "c25a8d75c439b4f45fc0b542fbdb75ff"
server_key_iv: "69e54f282dd67dfe941f276ee04153d9"
server_key_version: "encryptionkey@0"
I20260810 14:36:36.201658 8546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 14:36:36.204838 8571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:36.206101 8546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:36:36.206266 8546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "26ea528195824c24a58856832c8ae88c"
format_stamp: "Formatted at 2026-08-10 14:36:36 on dist-test-slave-q9h9"
server_key: "c25a8d75c439b4f45fc0b542fbdb75ff"
server_key_iv: "69e54f282dd67dfe941f276ee04153d9"
server_key_version: "encryptionkey@0"
I20260810 14:36:36.206444 8546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:36.211972 8546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:36.215864 8546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:36.216166 8546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:36.234265 8546 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37255
I20260810 14:36:36.234287 8663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37255 every 8 connection(s)
I20260810 14:36:36.235707 8546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:36.239256 8664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:36.242522 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 8546
I20260810 14:36:36.242678 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:36.243114 6331 external_mini_cluster.cc:1428] Setting key e870a75fee139ede75ea9f68d1f15fd5
I20260810 14:36:36.251322 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:36.244785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44523 (local address 127.0.0.1:37255)
0810 14:36:36.245606 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:36.245620 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:36.245683 (+ 63us) server_negotiation.cc:407] Connection header received
0810 14:36:36.247638 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:36.247676 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:36.248108 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:36.248525 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:36.249469 (+ 944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:36.249478 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:36.249494 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:36.249523 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:36.249588 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:36.249611 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:36.249848 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:36.249984 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 14:36:36.250184 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":330,"threads_started":1}
I20260810 14:36:36.252081 8664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c: Bootstrap starting.
I20260810 14:36:36.254637 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:36.255227 8664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:36.256412 8664 log.cc:824] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:36.259519 8664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c: No bootstrap required, opened a new log
I20260810 14:36:36.261616 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:36.259506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37473 (local address 127.0.0.1:37255)
0810 14:36:36.259674 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:36.259679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:36.259988 (+ 309us) server_negotiation.cc:407] Connection header received
0810 14:36:36.260089 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:36.260094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:36.260154 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:36.260258 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:36.261021 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:36.261027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:36.261035 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:36.261053 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:36.261100 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:36.261123 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:36.261241 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:36.261291 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 14:36:36.261349 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:36:36.267237 8664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea528195824c24a58856832c8ae88c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37255 } }
I20260810 14:36:36.267664 8664 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:36.267772 8664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ea528195824c24a58856832c8ae88c, State: Initialized, Role: FOLLOWER
I20260810 14:36:36.268323 8664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea528195824c24a58856832c8ae88c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37255 } }
I20260810 14:36:36.268535 8664 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:36.268663 8664 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:36.268801 8664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:36.270958 8664 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea528195824c24a58856832c8ae88c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37255 } }
I20260810 14:36:36.271411 8664 leader_election.cc:302] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26ea528195824c24a58856832c8ae88c; no voters:
I20260810 14:36:36.271955 8664 leader_election.cc:288] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:36.272071 8677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:36.272677 8677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [term 1 LEADER]: Becoming Leader. State: Replica: 26ea528195824c24a58856832c8ae88c, State: Running, Role: LEADER
I20260810 14:36:36.273159 8677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea528195824c24a58856832c8ae88c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37255 } }
I20260810 14:36:36.274003 8664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:36.275967 8679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26ea528195824c24a58856832c8ae88c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea528195824c24a58856832c8ae88c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37255 } } }
I20260810 14:36:36.276203 8683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26ea528195824c24a58856832c8ae88c. Latest consensus state: current_term: 1 leader_uuid: "26ea528195824c24a58856832c8ae88c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26ea528195824c24a58856832c8ae88c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37255 } } }
I20260810 14:36:36.276448 8679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:36.276468 8683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:36.281489 8704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:36.285553 8704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:36.293813 8704 catalog_manager.cc:1347] Generated new cluster ID: 4d348726c06b48108a285d99fa67d761
I20260810 14:36:36.293922 8704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:36.310041 8704 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:36.310525 8704 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:36.319702 8704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26ea528195824c24a58856832c8ae88c: Generated new TSK 0
I20260810 14:36:36.320488 8704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:36.332521 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8546
W20260810 14:36:36.333685 8656 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44523
W20260810 14:36:36.334425 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8546: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:36:36.334556 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 8546
2026-08-10T14:36:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:36 dist-test-slave-q9h9 krb5kdc[8726](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:36 dist-test-slave-q9h9 krb5kdc[8726](info): set up 2 sockets
Aug 10 14:36:36 dist-test-slave-q9h9 krb5kdc[8726](info): commencing operation
krb5kdc: starting...
W20260810 14:36:37.578706 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.003s sys 0.034s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:37 dist-test-slave-q9h9 krb5kdc[8726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:36:37.852284 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44073
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:38.173038 8755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:38.173323 8755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:38.173465 8755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:38.180493 8755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:38.180616 8755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:38.180693 8755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:38.180775 8755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:38.180835 8755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:38.190284 8755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:38.191440 8755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:38.192656 8755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:38.203474 8763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:38.203542 8762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:38.203550 8765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:38.204349 8755 server_base.cc:1034] running on GCE node
I20260810 14:36:38.205032 8755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:38.206429 8755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:38.207616 8755 hybrid_clock.cc:648] HybridClock initialized: now 1786372598207569 us; error 44 us; skew 500 ppm
I20260810 14:36:38.207940 8755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:38 dist-test-slave-q9h9 krb5kdc[8726](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372598, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:38.210999 8755 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:36:38.212340 8755 webserver.cc:460] Webserver started at http://127.0.0.1:45935/ using document root <none> and password file <none>
I20260810 14:36:38.212966 8755 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:38.213066 8755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:38.213307 8755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:38.216219 8755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "54d73562c4c140fbacd41873883d58a9"
format_stamp: "Formatted at 2026-08-10 14:36:38 on dist-test-slave-q9h9"
server_key: "bf46ed801902afcde43d4b39ab6ac602"
server_key_iv: "f925a1db7b9b3acf00a174aebde8b0b0"
server_key_version: "encryptionkey@0"
I20260810 14:36:38.216918 8755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "54d73562c4c140fbacd41873883d58a9"
format_stamp: "Formatted at 2026-08-10 14:36:38 on dist-test-slave-q9h9"
server_key: "bf46ed801902afcde43d4b39ab6ac602"
server_key_iv: "f925a1db7b9b3acf00a174aebde8b0b0"
server_key_version: "encryptionkey@0"
I20260810 14:36:38.221752 8755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 14:36:38.225134 8779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:38.226449 8755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 14:36:38.226610 8755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "54d73562c4c140fbacd41873883d58a9"
format_stamp: "Formatted at 2026-08-10 14:36:38 on dist-test-slave-q9h9"
server_key: "bf46ed801902afcde43d4b39ab6ac602"
server_key_iv: "f925a1db7b9b3acf00a174aebde8b0b0"
server_key_version: "encryptionkey@0"
I20260810 14:36:38.226791 8755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:38.246556 8755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:38.250442 8755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:38.250710 8755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:38.268011 8755 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44073
I20260810 14:36:38.268087 8880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44073 every 8 connection(s)
I20260810 14:36:38.269280 8755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:38.273036 8881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:38.277213 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 8755
I20260810 14:36:38.277415 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:38.278085 6331 external_mini_cluster.cc:1428] Setting key 956cc7aa332885e7ce1761138140ec28
I20260810 14:36:38.286250 8881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9: Bootstrap starting.
I20260810 14:36:38.290107 8881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:38.291404 8881 log.cc:824] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9: Log is configured to *not* fsync() on all Append() calls
Aug 10 14:36:38 dist-test-slave-q9h9 krb5kdc[8726](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 14:36:38.294299 8881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9: No bootstrap required, opened a new log
I20260810 14:36:38.300566 8884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:38.279808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44233 (local address 127.0.0.1:44073)
0810 14:36:38.280557 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:38.280567 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:38.280605 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:36:38.282213 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:38.282235 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:38.282604 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:38.282911 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:38.284857 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:38.286143 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:38.288062 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:38.288723 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:38.288841 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:38.293734 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:38.293742 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:38.293756 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:38.293788 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:38.296858 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:38.297487 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:38.297497 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:38.297503 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:38.297676 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:38.298060 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:38.298064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:38.298066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:38.298553 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:38.298858 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:38.299265 (+ 407us) server_negotiation.cc:299] Negotiation successful
0810 14:36:38.299658 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":359,"threads_started":1}
I20260810 14:36:38.302692 8881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d73562c4c140fbacd41873883d58a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260810 14:36:38.303061 8881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:38.303174 8881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d73562c4c140fbacd41873883d58a9, State: Initialized, Role: FOLLOWER
I20260810 14:36:38.303736 8881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d73562c4c140fbacd41873883d58a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260810 14:36:38.303954 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:38.303974 8881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:38.304101 8881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:38.304265 8881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:38.306573 8881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d73562c4c140fbacd41873883d58a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260810 14:36:38.307108 8881 leader_election.cc:302] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54d73562c4c140fbacd41873883d58a9; no voters:
I20260810 14:36:38.307724 8881 leader_election.cc:288] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:38.307843 8895 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:38.309998 8881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:38.310320 8895 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [term 1 LEADER]: Becoming Leader. State: Replica: 54d73562c4c140fbacd41873883d58a9, State: Running, Role: LEADER
I20260810 14:36:38.310992 8895 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d73562c4c140fbacd41873883d58a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } }
I20260810 14:36:38.314471 8896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54d73562c4c140fbacd41873883d58a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d73562c4c140fbacd41873883d58a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } } }
I20260810 14:36:38.314993 8896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:38.320308 8899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54d73562c4c140fbacd41873883d58a9. Latest consensus state: current_term: 1 leader_uuid: "54d73562c4c140fbacd41873883d58a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d73562c4c140fbacd41873883d58a9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44073 } } }
I20260810 14:36:38.321246 8899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:38.324712 8884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:38.308644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45295 (local address 127.0.0.1:44073)
0810 14:36:38.308791 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:38.308796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:38.309147 (+ 351us) server_negotiation.cc:407] Connection header received
0810 14:36:38.309301 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:38.309306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:38.309368 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:38.309500 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:38.311046 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:38.312113 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:38.313857 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:38.314542 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:38.314685 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:38.317955 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:38.317962 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:38.317968 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:38.318006 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:38.321314 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:38.322097 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:38.322103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:38.322106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:38.322253 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:38.322929 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:38.322934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:38.322937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:38.323326 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:38.323541 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:38.323948 (+ 407us) server_negotiation.cc:299] Negotiation successful
0810 14:36:38.324319 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 14:36:38.326488 8919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:36:38.326583 8919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:36:38.326689 8913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:38.328223 8913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:38.335990 8913 catalog_manager.cc:1347] Generated new cluster ID: 9c1b0ad15c484639945b7f5f09e6db5d
I20260810 14:36:38.336087 8913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:38.353677 8913 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:38.354887 8913 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:38.359599 8913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54d73562c4c140fbacd41873883d58a9: Generated new TSK 0
I20260810 14:36:38.360478 8913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:38.365289 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8755
I20260810 14:36:38.366704 8875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:38.453619 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 8755
2026-08-10T14:36:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2105 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:38 dist-test-slave-q9h9 krb5kdc[8928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:38 dist-test-slave-q9h9 krb5kdc[8928](info): set up 2 sockets
Aug 10 14:36:38 dist-test-slave-q9h9 krb5kdc[8928](info): commencing operation
krb5kdc: starting...
W20260810 14:36:39.689734 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.196s user 0.005s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:39 dist-test-slave-q9h9 krb5kdc[8928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372599, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:36:39.938694 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38507
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:40.257731 8950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:40.257992 8950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:40.258127 8950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:40.265055 8950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:40.265148 8950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:40.265215 8950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:40.265305 8950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:40.265362 8950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:40.274552 8950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:40.275728 8950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:40.276930 8950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:40.287874 8958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:40.287927 8959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:40.287961 8961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:40.288678 8950 server_base.cc:1034] running on GCE node
I20260810 14:36:40.289427 8950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:40.290776 8950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:40.291980 8950 hybrid_clock.cc:648] HybridClock initialized: now 1786372600291932 us; error 39 us; skew 500 ppm
I20260810 14:36:40.292299 8950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:40 dist-test-slave-q9h9 krb5kdc[8928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372600, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:40.298516 8950 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:36:40.299930 8950 webserver.cc:460] Webserver started at http://127.0.0.1:41375/ using document root <none> and password file <none>
I20260810 14:36:40.300554 8950 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:40.300673 8950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:40.301016 8950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:40.303853 8950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "87b1280caa9e49fb99391f27eb4f8eb3"
format_stamp: "Formatted at 2026-08-10 14:36:40 on dist-test-slave-q9h9"
server_key: "46f683025e23c24baa57963505f9513f"
server_key_iv: "42ce17912ea3cd39fee769c8611437f3"
server_key_version: "encryptionkey@0"
I20260810 14:36:40.304569 8950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "87b1280caa9e49fb99391f27eb4f8eb3"
format_stamp: "Formatted at 2026-08-10 14:36:40 on dist-test-slave-q9h9"
server_key: "46f683025e23c24baa57963505f9513f"
server_key_iv: "42ce17912ea3cd39fee769c8611437f3"
server_key_version: "encryptionkey@0"
I20260810 14:36:40.309399 8950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:36:40.312674 8970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:40.313942 8950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:36:40.314121 8950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "87b1280caa9e49fb99391f27eb4f8eb3"
format_stamp: "Formatted at 2026-08-10 14:36:40 on dist-test-slave-q9h9"
server_key: "46f683025e23c24baa57963505f9513f"
server_key_iv: "42ce17912ea3cd39fee769c8611437f3"
server_key_version: "encryptionkey@0"
I20260810 14:36:40.314286 8950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:40.327224 8950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:40.331024 8950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:40.331274 8950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:40.348315 8950 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38507
I20260810 14:36:40.348357 9061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38507 every 8 connection(s)
I20260810 14:36:40.349522 8950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:40.351214 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 8950
I20260810 14:36:40.351409 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:40.351960 6331 external_mini_cluster.cc:1428] Setting key 6cdca9287409e861807dbc1f2fd37b15
I20260810 14:36:40.354075 9062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:40.364441 9065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:40.353614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53847 (local address 127.0.0.1:38507)
0810 14:36:40.354385 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:40.354395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:40.354434 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:36:40.355989 (+ 1555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:40.356010 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:40.356368 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:40.356695 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:40.357244 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:40.358616 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:40.360462 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:40.361074 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:40.361187 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:40.362364 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:40.362375 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:40.362386 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:40.362913 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:40.362987 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:40.363011 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:40.363101 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:40.363333 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 14:36:40.363582 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":369,"threads_started":1}
I20260810 14:36:40.365942 9062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3: Bootstrap starting.
I20260810 14:36:40.367923 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:40.369028 9062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:40.369973 9062 log.cc:824] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:40.372848 9062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3: No bootstrap required, opened a new log
I20260810 14:36:40.378226 9065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:40.372199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38763 (local address 127.0.0.1:38507)
0810 14:36:40.372385 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:40.372390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:40.372717 (+ 327us) server_negotiation.cc:407] Connection header received
0810 14:36:40.372830 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:40.372838 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:40.372910 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:40.373008 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:40.373432 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:40.374461 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:40.375760 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:40.376464 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:40.376596 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:40.377473 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:40.377480 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:40.377484 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:40.377503 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:40.377551 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:40.377570 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:40.377677 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:40.377775 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 14:36:40.377867 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:36:40.381450 9062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1280caa9e49fb99391f27eb4f8eb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38507 } }
I20260810 14:36:40.381779 9062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:40.381850 9062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87b1280caa9e49fb99391f27eb4f8eb3, State: Initialized, Role: FOLLOWER
I20260810 14:36:40.382344 9062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1280caa9e49fb99391f27eb4f8eb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38507 } }
I20260810 14:36:40.382555 9062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:40.382655 9062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:40.382778 9062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:40.384959 9062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1280caa9e49fb99391f27eb4f8eb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38507 } }
I20260810 14:36:40.385390 9062 leader_election.cc:302] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 87b1280caa9e49fb99391f27eb4f8eb3; no voters:
I20260810 14:36:40.385901 9062 leader_election.cc:288] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:40.386046 9078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:40.386699 9078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [term 1 LEADER]: Becoming Leader. State: Replica: 87b1280caa9e49fb99391f27eb4f8eb3, State: Running, Role: LEADER
I20260810 14:36:40.387212 9078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1280caa9e49fb99391f27eb4f8eb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38507 } }
I20260810 14:36:40.387838 9062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:40.389822 9079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "87b1280caa9e49fb99391f27eb4f8eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1280caa9e49fb99391f27eb4f8eb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38507 } } }
I20260810 14:36:40.390254 9079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:40.389828 9080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87b1280caa9e49fb99391f27eb4f8eb3. Latest consensus state: current_term: 1 leader_uuid: "87b1280caa9e49fb99391f27eb4f8eb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "87b1280caa9e49fb99391f27eb4f8eb3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38507 } } }
I20260810 14:36:40.391716 9080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:40.394331 9085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:40.399199 9085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:40.407985 9085 catalog_manager.cc:1347] Generated new cluster ID: f88fa83570b24ee2ae7c06586dcd0083
I20260810 14:36:40.408093 9085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:40.418238 9085 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:40.419934 9085 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:40.431345 9085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 87b1280caa9e49fb99391f27eb4f8eb3: Generated new TSK 0
I20260810 14:36:40.432246 9085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:40.447973 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8950
W20260810 14:36:40.449038 9055 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53847
W20260810 14:36:40.449653 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 8950: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:36:40.449785 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 8950
2026-08-10T14:36:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:40 dist-test-slave-q9h9 krb5kdc[9109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:40 dist-test-slave-q9h9 krb5kdc[9109](info): set up 2 sockets
Aug 10 14:36:40 dist-test-slave-q9h9 krb5kdc[9109](info): commencing operation
krb5kdc: starting...
W20260810 14:36:41.676273 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:41 dist-test-slave-q9h9 krb5kdc[9109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:36:41.938668 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:42.254794 9132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:42.255055 9132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:42.255201 9132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:42.262190 9132 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:42.262288 9132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:42.262362 9132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:42.262444 9132 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:42.262501 9132 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:42.271829 9132 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:42.272943 9132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:42.274232 9132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:42.285369 9144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:42.285328 9142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:42.285866 9140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:42.286113 9132 server_base.cc:1034] running on GCE node
I20260810 14:36:42.286876 9132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:42.288219 9132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:42.289423 9132 hybrid_clock.cc:648] HybridClock initialized: now 1786372602289375 us; error 45 us; skew 500 ppm
I20260810 14:36:42.289752 9132 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:42 dist-test-slave-q9h9 krb5kdc[9109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372602, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:42.292860 9132 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:36:42.294282 9132 webserver.cc:460] Webserver started at http://127.0.0.1:41727/ using document root <none> and password file <none>
I20260810 14:36:42.294937 9132 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:42.295054 9132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:42.295338 9132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:42.298223 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "73e81638d9d943eb882fed9c7b1f28d5"
format_stamp: "Formatted at 2026-08-10 14:36:42 on dist-test-slave-q9h9"
server_key: "6436f15131ccdd30d23ab25600910a9e"
server_key_iv: "8e775f40a6f052c6db18b1242541a003"
server_key_version: "encryptionkey@0"
I20260810 14:36:42.298949 9132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "73e81638d9d943eb882fed9c7b1f28d5"
format_stamp: "Formatted at 2026-08-10 14:36:42 on dist-test-slave-q9h9"
server_key: "6436f15131ccdd30d23ab25600910a9e"
server_key_iv: "8e775f40a6f052c6db18b1242541a003"
server_key_version: "encryptionkey@0"
I20260810 14:36:42.303812 9132 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 14:36:42.307080 9155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:42.308261 9132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 14:36:42.308406 9132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "73e81638d9d943eb882fed9c7b1f28d5"
format_stamp: "Formatted at 2026-08-10 14:36:42 on dist-test-slave-q9h9"
server_key: "6436f15131ccdd30d23ab25600910a9e"
server_key_iv: "8e775f40a6f052c6db18b1242541a003"
server_key_version: "encryptionkey@0"
I20260810 14:36:42.308573 9132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:42.313931 9132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:42.317430 9132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:42.317694 9132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:42.334326 9132 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46335
I20260810 14:36:42.334378 9241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46335 every 8 connection(s)
I20260810 14:36:42.335564 9132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:42.338717 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 9132
I20260810 14:36:42.338877 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:42.339071 9243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:42.339303 6331 external_mini_cluster.cc:1428] Setting key 4e1cdb7b1be6f71af810987c2abb20b4
I20260810 14:36:42.347400 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:42.340861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57687 (local address 127.0.0.1:46335)
0810 14:36:42.341600 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:42.341615 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:42.341678 (+ 63us) server_negotiation.cc:407] Connection header received
0810 14:36:42.343699 (+ 2021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:42.343725 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:42.344121 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:42.344527 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:42.345901 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:42.345913 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:42.345926 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:42.345959 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:42.346076 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:42.346110 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:42.346287 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:42.346397 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 14:36:42.346554 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":290,"threads_started":1}
I20260810 14:36:42.350710 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:42.352515 9243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5: Bootstrap starting.
I20260810 14:36:42.356252 9243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:42.357275 9243 log.cc:824] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:42.357990 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:42.355659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55445 (local address 127.0.0.1:46335)
0810 14:36:42.355829 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:42.355834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:42.356119 (+ 285us) server_negotiation.cc:407] Connection header received
0810 14:36:42.356285 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:42.356289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:42.356336 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:42.356433 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:42.357305 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:42.357310 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:42.357315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:42.357332 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:42.357374 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:42.357395 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:42.357493 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:42.357593 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 14:36:42.357645 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 14:36:42.359941 9243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5: No bootstrap required, opened a new log
I20260810 14:36:42.366818 9243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e81638d9d943eb882fed9c7b1f28d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } }
I20260810 14:36:42.367156 9243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:42.367249 9243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e81638d9d943eb882fed9c7b1f28d5, State: Initialized, Role: FOLLOWER
I20260810 14:36:42.367772 9243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e81638d9d943eb882fed9c7b1f28d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } }
I20260810 14:36:42.367981 9243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:42.368112 9243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:42.368258 9243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:42.370344 9243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e81638d9d943eb882fed9c7b1f28d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } }
I20260810 14:36:42.370779 9243 leader_election.cc:302] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73e81638d9d943eb882fed9c7b1f28d5; no voters:
I20260810 14:36:42.371343 9243 leader_election.cc:288] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:42.371433 9255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:42.372392 9255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [term 1 LEADER]: Becoming Leader. State: Replica: 73e81638d9d943eb882fed9c7b1f28d5, State: Running, Role: LEADER
I20260810 14:36:42.372874 9255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e81638d9d943eb882fed9c7b1f28d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } }
I20260810 14:36:42.373292 9243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:42.375347 9257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73e81638d9d943eb882fed9c7b1f28d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e81638d9d943eb882fed9c7b1f28d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } } }
I20260810 14:36:42.375335 9258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73e81638d9d943eb882fed9c7b1f28d5. Latest consensus state: current_term: 1 leader_uuid: "73e81638d9d943eb882fed9c7b1f28d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e81638d9d943eb882fed9c7b1f28d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46335 } } }
I20260810 14:36:42.375790 9258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:42.375790 9257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:42.377667 9264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:42.382885 9264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:42.391691 9264 catalog_manager.cc:1347] Generated new cluster ID: 63abcba6d8554d87a5f108a2c0711a64
I20260810 14:36:42.391789 9264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:42.405498 9264 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:42.405933 9264 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:42.418637 9264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73e81638d9d943eb882fed9c7b1f28d5: Generated new TSK 0
I20260810 14:36:42.419530 9264 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:42.435803 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9132
W20260810 14:36:42.437029 9234 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57687
W20260810 14:36:42.437696 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9132: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:36:42.437817 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 9132
2026-08-10T14:36:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:42 dist-test-slave-q9h9 krb5kdc[9290](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:42 dist-test-slave-q9h9 krb5kdc[9290](info): set up 2 sockets
Aug 10 14:36:42 dist-test-slave-q9h9 krb5kdc[9290](info): commencing operation
krb5kdc: starting...
W20260810 14:36:43.678188 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.195s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:43 dist-test-slave-q9h9 krb5kdc[9290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:36:43.940560 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:44.255574 9316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:44.255824 9316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:44.255921 9316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:44.262782 9316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:44.262874 9316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:44.262925 9316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:44.262982 9316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:44.263026 9316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:44.272171 9316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:44.273217 9316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:44.274420 9316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:44.285326 9326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:44.285256 9330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:44.285296 9328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:44.286147 9316 server_base.cc:1034] running on GCE node
I20260810 14:36:44.286800 9316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:44.288074 9316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:44.289261 9316 hybrid_clock.cc:648] HybridClock initialized: now 1786372604289224 us; error 33 us; skew 500 ppm
I20260810 14:36:44.289566 9316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:44 dist-test-slave-q9h9 krb5kdc[9290](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372604, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:44.292546 9316 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:36:44.293918 9316 webserver.cc:460] Webserver started at http://127.0.0.1:45461/ using document root <none> and password file <none>
I20260810 14:36:44.294576 9316 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:44.294679 9316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:44.294934 9316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:44.297662 9316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "3724eb3264e94899aba57d8ebdd855f2"
format_stamp: "Formatted at 2026-08-10 14:36:44 on dist-test-slave-q9h9"
server_key: "81c0b16c58cf460c7b98f7d5f03ae070"
server_key_iv: "dfb130db940f2d6a6fec997c0731840e"
server_key_version: "encryptionkey@0"
I20260810 14:36:44.298391 9316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "3724eb3264e94899aba57d8ebdd855f2"
format_stamp: "Formatted at 2026-08-10 14:36:44 on dist-test-slave-q9h9"
server_key: "81c0b16c58cf460c7b98f7d5f03ae070"
server_key_iv: "dfb130db940f2d6a6fec997c0731840e"
server_key_version: "encryptionkey@0"
I20260810 14:36:44.303253 9316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 14:36:44.306490 9337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:44.307695 9316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 14:36:44.307849 9316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "3724eb3264e94899aba57d8ebdd855f2"
format_stamp: "Formatted at 2026-08-10 14:36:44 on dist-test-slave-q9h9"
server_key: "81c0b16c58cf460c7b98f7d5f03ae070"
server_key_iv: "dfb130db940f2d6a6fec997c0731840e"
server_key_version: "encryptionkey@0"
I20260810 14:36:44.308010 9316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:44.327688 9316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:44.331525 9316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:44.331766 9316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:44.348992 9316 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34375
I20260810 14:36:44.349030 9429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34375 every 8 connection(s)
I20260810 14:36:44.350281 9316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:44.351928 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 9316
I20260810 14:36:44.352077 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:44.352496 6331 external_mini_cluster.cc:1428] Setting key abea9b4672e56c2651b2ddffda10ca5a
I20260810 14:36:44.354446 9430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 14:36:44 dist-test-slave-q9h9 krb5kdc[9290](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372603, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 14:36:44.367708 9430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2: Bootstrap starting.
I20260810 14:36:44.370973 9430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:44.372017 9430 log.cc:824] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:44.374897 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:44.354288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:34431 (local address 127.0.0.1:34375)
0810 14:36:44.355018 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:44.355028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:44.355068 (+ 40us) server_negotiation.cc:407] Connection header received
0810 14:36:44.356665 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:44.356684 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:44.357043 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:44.357354 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:44.359274 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:44.360597 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:44.362468 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:44.363080 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:44.363213 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:44.368104 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:44.368111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:44.368123 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:44.368162 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:44.370957 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:44.371710 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:44.371716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:44.371722 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:44.371879 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:44.372355 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:44.372359 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:44.372361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:44.372872 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:44.373196 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:44.373668 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 14:36:44.373938 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":344,"threads_started":1}
I20260810 14:36:44.374971 9430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2: No bootstrap required, opened a new log
I20260810 14:36:44.378412 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:44.383572 9430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3724eb3264e94899aba57d8ebdd855f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34375 } }
I20260810 14:36:44.383906 9430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:44.384009 9430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3724eb3264e94899aba57d8ebdd855f2, State: Initialized, Role: FOLLOWER
I20260810 14:36:44.384588 9430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3724eb3264e94899aba57d8ebdd855f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34375 } }
I20260810 14:36:44.384899 9430 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:44.385087 9430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:44.385300 9430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:44.388105 9430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3724eb3264e94899aba57d8ebdd855f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34375 } }
I20260810 14:36:44.388597 9430 leader_election.cc:302] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3724eb3264e94899aba57d8ebdd855f2; no voters:
I20260810 14:36:44.389178 9430 leader_election.cc:288] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:44.389308 9442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:44.389926 9442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [term 1 LEADER]: Becoming Leader. State: Replica: 3724eb3264e94899aba57d8ebdd855f2, State: Running, Role: LEADER
I20260810 14:36:44.390560 9442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3724eb3264e94899aba57d8ebdd855f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34375 } }
I20260810 14:36:44.392464 9430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:44.393210 9443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3724eb3264e94899aba57d8ebdd855f2. Latest consensus state: current_term: 1 leader_uuid: "3724eb3264e94899aba57d8ebdd855f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3724eb3264e94899aba57d8ebdd855f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34375 } } }
I20260810 14:36:44.393568 9443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:44.393553 9447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3724eb3264e94899aba57d8ebdd855f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3724eb3264e94899aba57d8ebdd855f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34375 } } }
I20260810 14:36:44.397841 9447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:44.399184 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:44.382470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:55355 (local address 127.0.0.1:34375)
0810 14:36:44.382607 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:44.382611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:44.383044 (+ 433us) server_negotiation.cc:407] Connection header received
0810 14:36:44.383160 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:44.383166 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:44.383245 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:44.383359 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:44.384689 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:44.385804 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:44.387059 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:44.387797 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:44.387933 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:44.390641 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:44.390649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:44.390655 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:44.390699 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:44.394657 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:44.395796 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:44.395803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:44.395807 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:44.396023 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:44.396699 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:44.396704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:44.396707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:44.397121 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:44.397434 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:44.398514 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0810 14:36:44.398713 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:36:44.401202 9450 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:44.405483 9450 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:44.412864 9450 catalog_manager.cc:1347] Generated new cluster ID: d8043e06d3514a3b99c6476b7da94c5d
I20260810 14:36:44.412973 9450 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:44.432063 9450 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:44.433768 9450 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:44.446501 9450 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3724eb3264e94899aba57d8ebdd855f2: Generated new TSK 0
I20260810 14:36:44.447397 9450 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:44.460893 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9316
I20260810 14:36:44.461925 9421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:44.551587 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 9316
W20260810 14:36:44.573166 9288 connection.cc:535] client connection to 127.0.0.1:34375 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34375): Connection reset by peer (error 104)
2026-08-10T14:36:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2108 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:44 dist-test-slave-q9h9 krb5kdc[9483](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:44 dist-test-slave-q9h9 krb5kdc[9483](info): set up 2 sockets
Aug 10 14:36:44 dist-test-slave-q9h9 krb5kdc[9483](info): commencing operation
krb5kdc: starting...
W20260810 14:36:45.780153 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.189s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:45 dist-test-slave-q9h9 krb5kdc[9483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:36:46.051111 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:46.377543 9507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:46.377816 9507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:46.377951 9507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:46.384829 9507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:46.384936 9507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:46.385010 9507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:46.385087 9507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:46.385167 9507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:46.396435 9507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:46.397862 9507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:46.399240 9507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:46.411787 9514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:46.414497 9520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:46.415352 9512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:46.415843 9507 server_base.cc:1034] running on GCE node
I20260810 14:36:46.416770 9507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:46.418540 9507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:46.419788 9507 hybrid_clock.cc:648] HybridClock initialized: now 1786372606419727 us; error 52 us; skew 500 ppm
I20260810 14:36:46.420198 9507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:46 dist-test-slave-q9h9 krb5kdc[9483](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:46.424217 9507 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:36:46.426489 9507 webserver.cc:460] Webserver started at http://127.0.0.1:37083/ using document root <none> and password file <none>
I20260810 14:36:46.427251 9507 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:46.427371 9507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:46.427649 9507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:46.431226 9507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "95a517ff61ed4f2bbc5c4919ef17749d"
format_stamp: "Formatted at 2026-08-10 14:36:46 on dist-test-slave-q9h9"
server_key: "6b7dcf592e7bab407bab86163e0ef7a4"
server_key_iv: "bf14127931260365aa323e945f19854b"
server_key_version: "encryptionkey@0"
I20260810 14:36:46.432158 9507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "95a517ff61ed4f2bbc5c4919ef17749d"
format_stamp: "Formatted at 2026-08-10 14:36:46 on dist-test-slave-q9h9"
server_key: "6b7dcf592e7bab407bab86163e0ef7a4"
server_key_iv: "bf14127931260365aa323e945f19854b"
server_key_version: "encryptionkey@0"
I20260810 14:36:46.437868 9507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 14:36:46.441293 9532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:46.442664 9507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:46.442823 9507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "95a517ff61ed4f2bbc5c4919ef17749d"
format_stamp: "Formatted at 2026-08-10 14:36:46 on dist-test-slave-q9h9"
server_key: "6b7dcf592e7bab407bab86163e0ef7a4"
server_key_iv: "bf14127931260365aa323e945f19854b"
server_key_version: "encryptionkey@0"
I20260810 14:36:46.442986 9507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:46.478086 9507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:46.481611 9507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:46.481859 9507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:46.498867 9507 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34791
I20260810 14:36:46.498912 9622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34791 every 8 connection(s)
I20260810 14:36:46.500087 9507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:46.503670 9628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:46.507619 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 9507
I20260810 14:36:46.507782 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:46.508203 6331 external_mini_cluster.cc:1428] Setting key 4157e5730451816a5181ac3c1424dd8e
I20260810 14:36:46.515203 9628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d: Bootstrap starting.
I20260810 14:36:46.518778 9628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:46.520059 9628 log.cc:824] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:46.523058 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:46.509974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:33719 (local address 127.0.0.1:34791)
0810 14:36:46.510832 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:46.510850 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:46.511192 (+ 342us) server_negotiation.cc:407] Connection header received
0810 14:36:46.513700 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:46.513728 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:46.514213 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:46.514625 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:46.515135 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:46.517070 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:46.518377 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:46.519397 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:46.519578 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:46.520631 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:46.520642 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:46.520660 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:46.521190 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:46.521276 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:46.521307 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:46.521450 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:46.521700 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 14:36:46.521851 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":354,"threads_started":1}
I20260810 14:36:46.523103 9628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d: No bootstrap required, opened a new log
I20260810 14:36:46.526628 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:46.532143 9628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a517ff61ed4f2bbc5c4919ef17749d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34791 } }
I20260810 14:36:46.532582 9628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:46.532676 9628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95a517ff61ed4f2bbc5c4919ef17749d, State: Initialized, Role: FOLLOWER
I20260810 14:36:46.533308 9628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a517ff61ed4f2bbc5c4919ef17749d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34791 } }
I20260810 14:36:46.533589 9628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:46.533738 9628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:46.533907 9628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:46.536413 9628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a517ff61ed4f2bbc5c4919ef17749d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34791 } }
I20260810 14:36:46.536861 9628 leader_election.cc:302] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95a517ff61ed4f2bbc5c4919ef17749d; no voters:
I20260810 14:36:46.537515 9628 leader_election.cc:288] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:46.537624 9650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:46.537868 9632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:46.531525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:33903 (local address 127.0.0.1:34791)
0810 14:36:46.531682 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:46.531687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:46.532101 (+ 414us) server_negotiation.cc:407] Connection header received
0810 14:36:46.532235 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:46.532239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:46.532308 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:46.532410 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:46.532876 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:46.533961 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:46.535218 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:46.535913 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:46.536029 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:46.536991 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:46.536997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:46.537002 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:46.537022 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:46.537064 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:46.537084 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:46.537202 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:46.537374 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 14:36:46.537427 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:36:46.539286 9650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [term 1 LEADER]: Becoming Leader. State: Replica: 95a517ff61ed4f2bbc5c4919ef17749d, State: Running, Role: LEADER
I20260810 14:36:46.539863 9628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:46.539844 9650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a517ff61ed4f2bbc5c4919ef17749d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34791 } }
I20260810 14:36:46.542846 9652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95a517ff61ed4f2bbc5c4919ef17749d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a517ff61ed4f2bbc5c4919ef17749d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34791 } } }
I20260810 14:36:46.542798 9654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95a517ff61ed4f2bbc5c4919ef17749d. Latest consensus state: current_term: 1 leader_uuid: "95a517ff61ed4f2bbc5c4919ef17749d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95a517ff61ed4f2bbc5c4919ef17749d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34791 } } }
I20260810 14:36:46.543272 9654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:46.543286 9652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:46.547930 9664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:46.552805 9664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:46.561440 9664 catalog_manager.cc:1347] Generated new cluster ID: 1dae5c2fa2cb4cd4a9d12091b0a92617
I20260810 14:36:46.561532 9664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:46.577169 9664 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:46.578383 9664 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:46.590808 9664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95a517ff61ed4f2bbc5c4919ef17749d: Generated new TSK 0
I20260810 14:36:46.591566 9664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:46.603616 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9507
W20260810 14:36:46.604729 9617 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.62:33719
W20260810 14:36:46.605476 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9507: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:36:46.605624 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 9507
W20260810 14:36:46.631044 9478 connection.cc:535] client connection to 127.0.0.1:34791 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34791): Connection reset by peer (error 104)
2026-08-10T14:36:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2057 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:46 dist-test-slave-q9h9 krb5kdc[9683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:46 dist-test-slave-q9h9 krb5kdc[9683](info): set up 2 sockets
Aug 10 14:36:46 dist-test-slave-q9h9 krb5kdc[9683](info): commencing operation
krb5kdc: starting...
W20260810 14:36:47.835219 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.001s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:48 dist-test-slave-q9h9 krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372608, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:36:48.103204 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39579
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39579
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:48.422590 9706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:48.422857 9706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:48.422995 9706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:48.429802 9706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:48.429908 9706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:48.429986 9706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:48.430091 9706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:48.430156 9706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:48.439467 9706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:48.440569 9706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:48.441751 9706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:48.452791 9716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:48.452824 9717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:48.452735 9720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:48.453539 9706 server_base.cc:1034] running on GCE node
I20260810 14:36:48.454269 9706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:48.455535 9706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:48.456748 9706 hybrid_clock.cc:648] HybridClock initialized: now 1786372608456696 us; error 47 us; skew 500 ppm
I20260810 14:36:48.457059 9706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:48 dist-test-slave-q9h9 krb5kdc[9683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372608, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:48.459957 9706 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:36:48.461313 9706 webserver.cc:460] Webserver started at http://127.0.0.1:45813/ using document root <none> and password file <none>
I20260810 14:36:48.461923 9706 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:48.462056 9706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:48.462334 9706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:48.465050 9706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "6c2e3112c6584583b5b6d319461d849a"
format_stamp: "Formatted at 2026-08-10 14:36:48 on dist-test-slave-q9h9"
server_key: "70116b5d951db96d1586f1a59ed22820"
server_key_iv: "84b3a25e7adf56b50702d9d1bc6f6807"
server_key_version: "encryptionkey@0"
I20260810 14:36:48.465746 9706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "6c2e3112c6584583b5b6d319461d849a"
format_stamp: "Formatted at 2026-08-10 14:36:48 on dist-test-slave-q9h9"
server_key: "70116b5d951db96d1586f1a59ed22820"
server_key_iv: "84b3a25e7adf56b50702d9d1bc6f6807"
server_key_version: "encryptionkey@0"
I20260810 14:36:48.470419 9706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 14:36:48.473498 9733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:48.474674 9706 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:36:48.474838 9706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "6c2e3112c6584583b5b6d319461d849a"
format_stamp: "Formatted at 2026-08-10 14:36:48 on dist-test-slave-q9h9"
server_key: "70116b5d951db96d1586f1a59ed22820"
server_key_iv: "84b3a25e7adf56b50702d9d1bc6f6807"
server_key_version: "encryptionkey@0"
I20260810 14:36:48.475003 9706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:48.480077 9706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:48.483811 9706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:48.484066 9706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:48.499706 9706 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39579
I20260810 14:36:48.499754 9818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39579 every 8 connection(s)
I20260810 14:36:48.500869 9706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:48.504225 9822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:48.506742 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 9706
I20260810 14:36:48.506910 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:48.507448 6331 external_mini_cluster.cc:1428] Setting key 5a3b4177bf3793473facdb8fb4f8020a
I20260810 14:36:48.514607 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:48.509132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:37211 (local address 127.0.0.1:39579)
0810 14:36:48.509899 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:48.509909 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:48.509948 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:36:48.511574 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:48.511596 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:48.511899 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:48.512217 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:48.513178 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:48.513184 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:48.513196 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:48.513210 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:48.513267 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:48.513285 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:48.513461 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:48.513560 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 14:36:48.513749 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":330,"threads_started":1}
I20260810 14:36:48.515391 9822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a: Bootstrap starting.
I20260810 14:36:48.517836 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:48.518399 9822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:48.519531 9822 log.cc:824] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:48.522455 9822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a: No bootstrap required, opened a new log
I20260810 14:36:48.524917 9829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:48.522700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:51725 (local address 127.0.0.1:39579)
0810 14:36:48.522867 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:48.522872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:48.523223 (+ 351us) server_negotiation.cc:407] Connection header received
0810 14:36:48.523341 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:48.523345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:48.523397 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:48.523505 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:48.524269 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:48.524274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:48.524278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:48.524301 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:48.524346 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:48.524365 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:48.524494 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:48.524533 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 14:36:48.524582 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:36:48.529951 9822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2e3112c6584583b5b6d319461d849a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39579 } }
I20260810 14:36:48.530293 9822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:48.530368 9822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2e3112c6584583b5b6d319461d849a, State: Initialized, Role: FOLLOWER
I20260810 14:36:48.530836 9822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2e3112c6584583b5b6d319461d849a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39579 } }
I20260810 14:36:48.531034 9822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:48.531126 9822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:48.531255 9822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:48.533430 9822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2e3112c6584583b5b6d319461d849a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39579 } }
I20260810 14:36:48.533842 9822 leader_election.cc:302] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c2e3112c6584583b5b6d319461d849a; no voters:
I20260810 14:36:48.534345 9822 leader_election.cc:288] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:48.534480 9839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:48.535115 9839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [term 1 LEADER]: Becoming Leader. State: Replica: 6c2e3112c6584583b5b6d319461d849a, State: Running, Role: LEADER
I20260810 14:36:48.535597 9839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2e3112c6584583b5b6d319461d849a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39579 } }
I20260810 14:36:48.536123 9822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:48.538195 9841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c2e3112c6584583b5b6d319461d849a. Latest consensus state: current_term: 1 leader_uuid: "6c2e3112c6584583b5b6d319461d849a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2e3112c6584583b5b6d319461d849a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39579 } } }
I20260810 14:36:48.538556 9841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:48.538930 9840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c2e3112c6584583b5b6d319461d849a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c2e3112c6584583b5b6d319461d849a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39579 } } }
I20260810 14:36:48.539315 9840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:48.540369 9847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:48.545714 9847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:48.553659 9847 catalog_manager.cc:1347] Generated new cluster ID: d9e5cd7f2d4c475f8f36d8d252ffc15d
I20260810 14:36:48.553751 9847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:48.573140 9847 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:48.573570 9847 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:48.584630 9847 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c2e3112c6584583b5b6d319461d849a: Generated new TSK 0
I20260810 14:36:48.585520 9847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:48.593253 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9706
W20260810 14:36:48.594342 9814 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.62:37211
W20260810 14:36:48.595528 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9706: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:36:48.595641 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 9706
2026-08-10T14:36:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:48 dist-test-slave-q9h9 krb5kdc[9870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:48 dist-test-slave-q9h9 krb5kdc[9870](info): set up 2 sockets
Aug 10 14:36:48 dist-test-slave-q9h9 krb5kdc[9870](info): commencing operation
krb5kdc: starting...
W20260810 14:36:49.844995 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.205s user 0.003s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:50 dist-test-slave-q9h9 krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:36:50.110678 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:50.430111 9897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:50.430390 9897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:50.430508 9897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:50.437441 9897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:50.437536 9897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:50.437605 9897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:50.437690 9897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:50.437745 9897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:50.447006 9897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:50.448136 9897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:50.449359 9897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:50.460403 9909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:50.460470 9906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:50.460403 9913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:50.461268 9897 server_base.cc:1034] running on GCE node
I20260810 14:36:50.461938 9897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:50.463243 9897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:50.464433 9897 hybrid_clock.cc:648] HybridClock initialized: now 1786372610464388 us; error 38 us; skew 500 ppm
I20260810 14:36:50.464756 9897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:50 dist-test-slave-q9h9 krb5kdc[9870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:50.467659 9897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:36:50.469030 9897 webserver.cc:460] Webserver started at http://127.0.0.1:36691/ using document root <none> and password file <none>
I20260810 14:36:50.469661 9897 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:50.469777 9897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:50.470096 9897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:50.472851 9897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "86cdbcdfdb344fd6bf3f8ef575aef2e0"
format_stamp: "Formatted at 2026-08-10 14:36:50 on dist-test-slave-q9h9"
server_key: "e02b28f09527b2c9999eba65d7ac8bb4"
server_key_iv: "9d1fc21185e0be75d2676b5fcbcfd57a"
server_key_version: "encryptionkey@0"
I20260810 14:36:50.473556 9897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "86cdbcdfdb344fd6bf3f8ef575aef2e0"
format_stamp: "Formatted at 2026-08-10 14:36:50 on dist-test-slave-q9h9"
server_key: "e02b28f09527b2c9999eba65d7ac8bb4"
server_key_iv: "9d1fc21185e0be75d2676b5fcbcfd57a"
server_key_version: "encryptionkey@0"
I20260810 14:36:50.478305 9897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260810 14:36:50.481503 9922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:50.482717 9897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:50.482868 9897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "86cdbcdfdb344fd6bf3f8ef575aef2e0"
format_stamp: "Formatted at 2026-08-10 14:36:50 on dist-test-slave-q9h9"
server_key: "e02b28f09527b2c9999eba65d7ac8bb4"
server_key_iv: "9d1fc21185e0be75d2676b5fcbcfd57a"
server_key_version: "encryptionkey@0"
I20260810 14:36:50.483031 9897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:50.497871 9897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:50.501247 9897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:50.501490 9897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:50.518358 9897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43851
I20260810 14:36:50.518411 10017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43851 every 8 connection(s)
I20260810 14:36:50.519544 9897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:50.522897 10018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:50.523787 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 9897
I20260810 14:36:50.523976 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:50.524557 6331 external_mini_cluster.cc:1428] Setting key ca0102dabf0d98e3b3b4904ffd86a19e
I20260810 14:36:50.535929 10018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0: Bootstrap starting.
Aug 10 14:36:50 dist-test-slave-q9h9 krb5kdc[9870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372610, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 14:36:50.539357 10018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:50.540701 10018 log.cc:824] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:50.543802 10018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0: No bootstrap required, opened a new log
I20260810 14:36:50.547477 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:50.526345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:33921 (local address 127.0.0.1:43851)
0810 14:36:50.527056 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:50.527073 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:50.527111 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:36:50.528763 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:50.528783 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:50.529139 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:50.529444 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:50.531182 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:50.532410 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:50.534162 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:50.534777 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:50.534905 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:50.539450 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:50.539462 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:50.539482 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:50.539528 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:50.543101 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:50.543850 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:50.543859 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:50.543866 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:50.544056 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:50.544504 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:50.544510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:50.544513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:50.545179 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:50.545511 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:50.545834 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 14:36:50.546190 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":312,"threads_started":1}
I20260810 14:36:50.551347 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:50.553095 10018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86cdbcdfdb344fd6bf3f8ef575aef2e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43851 } }
I20260810 14:36:50.553417 10018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:50.553509 10018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86cdbcdfdb344fd6bf3f8ef575aef2e0, State: Initialized, Role: FOLLOWER
I20260810 14:36:50.554118 10018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86cdbcdfdb344fd6bf3f8ef575aef2e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43851 } }
I20260810 14:36:50.554399 10018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:50.554535 10018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:50.554692 10018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:50.557117 10018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86cdbcdfdb344fd6bf3f8ef575aef2e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43851 } }
I20260810 14:36:50.557554 10018 leader_election.cc:302] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86cdbcdfdb344fd6bf3f8ef575aef2e0; no voters:
I20260810 14:36:50.558213 10018 leader_election.cc:288] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:50.558329 10038 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:50.560235 10018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:50.560788 10038 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [term 1 LEADER]: Becoming Leader. State: Replica: 86cdbcdfdb344fd6bf3f8ef575aef2e0, State: Running, Role: LEADER
I20260810 14:36:50.561362 10038 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86cdbcdfdb344fd6bf3f8ef575aef2e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43851 } }
I20260810 14:36:50.564173 10040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86cdbcdfdb344fd6bf3f8ef575aef2e0. Latest consensus state: current_term: 1 leader_uuid: "86cdbcdfdb344fd6bf3f8ef575aef2e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86cdbcdfdb344fd6bf3f8ef575aef2e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43851 } } }
I20260810 14:36:50.564535 10040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:50.568015 10039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86cdbcdfdb344fd6bf3f8ef575aef2e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86cdbcdfdb344fd6bf3f8ef575aef2e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43851 } } }
I20260810 14:36:50.568310 10039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:50.572568 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:50.555461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:55659 (local address 127.0.0.1:43851)
0810 14:36:50.555627 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:50.555632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:50.556073 (+ 441us) server_negotiation.cc:407] Connection header received
0810 14:36:50.556185 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:50.556190 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:50.556266 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:50.556363 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:50.557715 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:50.558896 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:50.560580 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:50.561298 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:50.561431 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:50.564783 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:50.564789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:50.564793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:50.564833 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:50.568794 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:50.569793 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:50.569799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:50.569803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:50.569951 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:50.570565 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:50.570570 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:50.570574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:50.570982 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:50.572000 (+ 1018us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:50.572175 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 14:36:50.572303 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:36:50.574477 10046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:50.579030 10046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:50.587020 10046 catalog_manager.cc:1347] Generated new cluster ID: 21a4dc62925c4008ba9b29290f692ff7
I20260810 14:36:50.587142 10046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:50.603751 10046 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:50.604894 10046 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:50.612447 10046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86cdbcdfdb344fd6bf3f8ef575aef2e0: Generated new TSK 0
I20260810 14:36:50.613330 10046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:50.625205 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 9897
I20260810 14:36:50.626441 10010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:50.719106 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 9897
W20260810 14:36:50.743690 9866 connection.cc:535] client connection to 127.0.0.1:43851 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43851): Connection reset by peer (error 104)
2026-08-10T14:36:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2126 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:50 dist-test-slave-q9h9 krb5kdc[10071](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:50 dist-test-slave-q9h9 krb5kdc[10071](info): set up 2 sockets
Aug 10 14:36:50 dist-test-slave-q9h9 krb5kdc[10071](info): commencing operation
krb5kdc: starting...
W20260810 14:36:51.976049 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.210s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:52 dist-test-slave-q9h9 krb5kdc[10071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:36:52.228613 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42521
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:52.545895 10090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:52.546178 10090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:52.546307 10090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:52.553299 10090 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:52.553391 10090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:52.553465 10090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:52.553541 10090 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:52.553599 10090 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:52.562932 10090 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42521
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:52.564044 10090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:52.565299 10090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:52.576238 10100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:52.576201 10106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:52.576274 10101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:52.577296 10090 server_base.cc:1034] running on GCE node
I20260810 14:36:52.577980 10090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:52.579283 10090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:52.580477 10090 hybrid_clock.cc:648] HybridClock initialized: now 1786372612580434 us; error 38 us; skew 500 ppm
I20260810 14:36:52.580794 10090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:52 dist-test-slave-q9h9 krb5kdc[10071](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:52.589284 10090 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:36:52.590852 10090 webserver.cc:460] Webserver started at http://127.0.0.1:33603/ using document root <none> and password file <none>
I20260810 14:36:52.591586 10090 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:52.591707 10090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:52.591993 10090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:52.594849 10090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "84df066dcadc49c7b571ecfbfa78e24d"
format_stamp: "Formatted at 2026-08-10 14:36:52 on dist-test-slave-q9h9"
server_key: "4c1a62be20d336b39f6e832368799df0"
server_key_iv: "42c1c4ddfff0ea769cad0883a7a10633"
server_key_version: "encryptionkey@0"
I20260810 14:36:52.595559 10090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "84df066dcadc49c7b571ecfbfa78e24d"
format_stamp: "Formatted at 2026-08-10 14:36:52 on dist-test-slave-q9h9"
server_key: "4c1a62be20d336b39f6e832368799df0"
server_key_iv: "42c1c4ddfff0ea769cad0883a7a10633"
server_key_version: "encryptionkey@0"
I20260810 14:36:52.600381 10090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:36:52.603627 10119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:52.604825 10090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:36:52.604967 10090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "84df066dcadc49c7b571ecfbfa78e24d"
format_stamp: "Formatted at 2026-08-10 14:36:52 on dist-test-slave-q9h9"
server_key: "4c1a62be20d336b39f6e832368799df0"
server_key_iv: "42c1c4ddfff0ea769cad0883a7a10633"
server_key_version: "encryptionkey@0"
I20260810 14:36:52.605129 10090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:52.624815 10090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:52.628737 10090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:52.628980 10090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:52.646219 10090 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42521
I20260810 14:36:52.646286 10201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42521 every 8 connection(s)
I20260810 14:36:52.647415 10090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:52.649860 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 10090
I20260810 14:36:52.650101 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:52.650725 6331 external_mini_cluster.cc:1428] Setting key 663048940af91c99b544a9094253b7da
I20260810 14:36:52.651479 10202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:52.664062 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:52.652444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:46887 (local address 127.0.0.1:42521)
0810 14:36:52.653184 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:52.653195 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:52.653234 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:36:52.654824 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:52.654846 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:52.655238 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:52.655554 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:52.656224 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:52.657531 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:52.659597 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:52.660272 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:52.660412 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:52.661775 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:52.661787 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:52.661800 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:52.662405 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:52.662480 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:52.662505 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:52.662635 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:52.662925 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 14:36:52.663092 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":323,"threads_started":1}
I20260810 14:36:52.665478 10202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d: Bootstrap starting.
I20260810 14:36:52.667658 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:52.668994 10202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:52.670310 10202 log.cc:824] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:52.673508 10202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d: No bootstrap required, opened a new log
I20260810 14:36:52.680711 10206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:52.673572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:44587 (local address 127.0.0.1:42521)
0810 14:36:52.673698 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:52.673701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:52.674171 (+ 470us) server_negotiation.cc:407] Connection header received
0810 14:36:52.674364 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:52.674367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:52.674422 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:52.674498 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:52.675177 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:52.675926 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:52.677836 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:52.678541 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:52.678659 (+ 118us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:52.679907 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:52.679912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:52.679915 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:52.679934 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:52.679970 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:52.679985 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:52.680081 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:52.680349 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 14:36:52.680409 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 14:36:52.683137 10202 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84df066dcadc49c7b571ecfbfa78e24d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260810 14:36:52.683483 10202 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:52.683585 10202 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84df066dcadc49c7b571ecfbfa78e24d, State: Initialized, Role: FOLLOWER
I20260810 14:36:52.684118 10202 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84df066dcadc49c7b571ecfbfa78e24d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260810 14:36:52.684351 10202 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:52.684484 10202 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:52.684646 10202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:52.686882 10202 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84df066dcadc49c7b571ecfbfa78e24d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260810 14:36:52.687386 10202 leader_election.cc:302] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84df066dcadc49c7b571ecfbfa78e24d; no voters:
I20260810 14:36:52.687901 10202 leader_election.cc:288] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:52.687996 10218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:52.688658 10218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [term 1 LEADER]: Becoming Leader. State: Replica: 84df066dcadc49c7b571ecfbfa78e24d, State: Running, Role: LEADER
I20260810 14:36:52.689194 10218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84df066dcadc49c7b571ecfbfa78e24d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } }
I20260810 14:36:52.689947 10202 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:52.691890 10222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84df066dcadc49c7b571ecfbfa78e24d. Latest consensus state: current_term: 1 leader_uuid: "84df066dcadc49c7b571ecfbfa78e24d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84df066dcadc49c7b571ecfbfa78e24d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } } }
I20260810 14:36:52.691921 10219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84df066dcadc49c7b571ecfbfa78e24d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84df066dcadc49c7b571ecfbfa78e24d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42521 } } }
I20260810 14:36:52.692348 10219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:52.693717 10222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:52.693985 10230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:52.698971 10230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:52.707828 10230 catalog_manager.cc:1347] Generated new cluster ID: 6b1cce5f07a440079ef9772cd97b9ff7
I20260810 14:36:52.707953 10230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:52.729434 10230 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:52.731139 10230 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:52.750202 10230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84df066dcadc49c7b571ecfbfa78e24d: Generated new TSK 0
I20260810 14:36:52.750897 10230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:52.759509 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10090
W20260810 14:36:52.760797 10197 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.62:46887
W20260810 14:36:52.761475 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10090: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:36:52.761623 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 10090
W20260810 14:36:52.789578 10069 connection.cc:535] client connection to 127.0.0.1:42521 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42521): Connection reset by peer (error 104)
2026-08-10T14:36:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2041 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:52 dist-test-slave-q9h9 krb5kdc[10256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:52 dist-test-slave-q9h9 krb5kdc[10256](info): set up 2 sockets
Aug 10 14:36:52 dist-test-slave-q9h9 krb5kdc[10256](info): commencing operation
krb5kdc: starting...
W20260810 14:36:54.020102 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:54 dist-test-slave-q9h9 krb5kdc[10256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372614, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 14:36:54.268523 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:54.585902 10273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:54.586205 10273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:54.586342 10273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:54.593708 10273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:54.593822 10273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:54.593895 10273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:54.593976 10273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:54.594065 10273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:54.603603 10273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42177
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:54.604761 10273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:54.605994 10273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:54.617200 10285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:54.617237 10282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:54.617149 10281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:54.618306 10273 server_base.cc:1034] running on GCE node
I20260810 14:36:54.619007 10273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:54.620294 10273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:54.621508 10273 hybrid_clock.cc:648] HybridClock initialized: now 1786372614621469 us; error 24 us; skew 500 ppm
I20260810 14:36:54.621835 10273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:54 dist-test-slave-q9h9 krb5kdc[10256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372614, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:54.624791 10273 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 14:36:54.626225 10273 webserver.cc:460] Webserver started at http://127.0.0.1:46503/ using document root <none> and password file <none>
I20260810 14:36:54.626817 10273 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:54.626917 10273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:54.627174 10273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:54.629904 10273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "9fa17bbdc2014cb9a12684f5ca4c8633"
format_stamp: "Formatted at 2026-08-10 14:36:54 on dist-test-slave-q9h9"
server_key: "db8a97a14eceedbc6ca91be25bb39d6b"
server_key_iv: "aebc78b41adb4ff723f6dc5240701959"
server_key_version: "encryptionkey@0"
I20260810 14:36:54.630725 10273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "9fa17bbdc2014cb9a12684f5ca4c8633"
format_stamp: "Formatted at 2026-08-10 14:36:54 on dist-test-slave-q9h9"
server_key: "db8a97a14eceedbc6ca91be25bb39d6b"
server_key_iv: "aebc78b41adb4ff723f6dc5240701959"
server_key_version: "encryptionkey@0"
I20260810 14:36:54.635560 10273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 14:36:54.638829 10293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:54.640069 10273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:36:54.640226 10273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "9fa17bbdc2014cb9a12684f5ca4c8633"
format_stamp: "Formatted at 2026-08-10 14:36:54 on dist-test-slave-q9h9"
server_key: "db8a97a14eceedbc6ca91be25bb39d6b"
server_key_iv: "aebc78b41adb4ff723f6dc5240701959"
server_key_version: "encryptionkey@0"
I20260810 14:36:54.640394 10273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:54.645938 10273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:54.649734 10273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:54.650000 10273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:54.667704 10273 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42177
I20260810 14:36:54.667748 10386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42177 every 8 connection(s)
I20260810 14:36:54.669065 10273 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:54.669477 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 10273
I20260810 14:36:54.669647 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:54.670266 6331 external_mini_cluster.cc:1428] Setting key f1a0bd8b64e4c796468331c87199b741
I20260810 14:36:54.673262 10387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:54.677635 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:54.672045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:34861 (local address 127.0.0.1:42177)
0810 14:36:54.672790 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:54.672801 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:54.672841 (+ 40us) server_negotiation.cc:407] Connection header received
0810 14:36:54.674523 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:54.674545 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:54.674871 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:54.675201 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:54.676147 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:54.676153 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:54.676165 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:54.676191 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:54.676259 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:54.676278 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:54.676472 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:54.676638 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 14:36:54.676800 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":330,"threads_started":1}
I20260810 14:36:54.681237 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:54.684491 10387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633: Bootstrap starting.
I20260810 14:36:54.687588 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:54.685439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.62:36783 (local address 127.0.0.1:42177)
0810 14:36:54.685594 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:54.685599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:54.685879 (+ 280us) server_negotiation.cc:407] Connection header received
0810 14:36:54.685996 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:54.686001 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:54.686097 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:54.686205 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:54.686997 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:54.687002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:54.687007 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:54.687056 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:54.687102 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:54.687124 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:54.687236 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:54.687281 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 14:36:54.687333 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:36:54.687951 10387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:54.689124 10387 log.cc:824] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:54.691578 10387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633: No bootstrap required, opened a new log
I20260810 14:36:54.698374 10387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa17bbdc2014cb9a12684f5ca4c8633" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42177 } }
I20260810 14:36:54.698693 10387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:54.698786 10387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fa17bbdc2014cb9a12684f5ca4c8633, State: Initialized, Role: FOLLOWER
I20260810 14:36:54.699270 10387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa17bbdc2014cb9a12684f5ca4c8633" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42177 } }
I20260810 14:36:54.699472 10387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:54.699630 10387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:54.699824 10387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:54.701824 10387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa17bbdc2014cb9a12684f5ca4c8633" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42177 } }
I20260810 14:36:54.702286 10387 leader_election.cc:302] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9fa17bbdc2014cb9a12684f5ca4c8633; no voters:
I20260810 14:36:54.702818 10387 leader_election.cc:288] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:54.702934 10408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:54.704015 10408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [term 1 LEADER]: Becoming Leader. State: Replica: 9fa17bbdc2014cb9a12684f5ca4c8633, State: Running, Role: LEADER
I20260810 14:36:54.704474 10408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa17bbdc2014cb9a12684f5ca4c8633" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42177 } }
I20260810 14:36:54.704756 10387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:54.707065 10411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9fa17bbdc2014cb9a12684f5ca4c8633. Latest consensus state: current_term: 1 leader_uuid: "9fa17bbdc2014cb9a12684f5ca4c8633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa17bbdc2014cb9a12684f5ca4c8633" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42177 } } }
I20260810 14:36:54.707125 10410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9fa17bbdc2014cb9a12684f5ca4c8633" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9fa17bbdc2014cb9a12684f5ca4c8633" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42177 } } }
I20260810 14:36:54.707621 10411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:54.707634 10410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:54.708545 10425 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:54.714609 10425 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:54.723367 10425 catalog_manager.cc:1347] Generated new cluster ID: 8ef393e854dd496cba7b5821bab247c2
I20260810 14:36:54.723469 10425 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:54.752743 10425 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:54.753088 10425 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:54.760479 10425 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9fa17bbdc2014cb9a12684f5ca4c8633: Generated new TSK 0
I20260810 14:36:54.761130 10425 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:54.770278 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10273
W20260810 14:36:54.771272 10381 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.62:34861
W20260810 14:36:54.772011 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10273: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:36:54.772145 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 10273
2026-08-10T14:36:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2010 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:54 dist-test-slave-q9h9 krb5kdc[10441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:54 dist-test-slave-q9h9 krb5kdc[10441](info): set up 2 sockets
Aug 10 14:36:54 dist-test-slave-q9h9 krb5kdc[10441](info): commencing operation
krb5kdc: starting...
W20260810 14:36:56.004621 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.186s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:56 dist-test-slave-q9h9 krb5kdc[10441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:36:56.254002 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:39455
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:34281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:39455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:56.580668 10467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:56.580919 10467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:56.581045 10467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:56.587915 10467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:56.588017 10467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:56.588095 10467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:56.588173 10467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:56.588231 10467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:56.597393 10467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:34281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:39455
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:39455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:56.598497 10467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:56.599658 10467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:56.610658 10476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:56.610608 10479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:56.610690 10483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:56.611517 10467 server_base.cc:1034] running on GCE node
I20260810 14:36:56.612226 10467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:56.613515 10467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:56.614735 10467 hybrid_clock.cc:648] HybridClock initialized: now 1786372616614672 us; error 37 us; skew 500 ppm
I20260810 14:36:56.615067 10467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:56 dist-test-slave-q9h9 krb5kdc[10441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372616, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:56.618124 10467 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:36:56.619549 10467 webserver.cc:460] Webserver started at http://127.6.46.254:46001/ using document root <none> and password file <none>
I20260810 14:36:56.620164 10467 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:56.620271 10467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:56.620532 10467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:56.623468 10467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "8ff7ec36375049d38bfc6db829d5a9a8"
format_stamp: "Formatted at 2026-08-10 14:36:56 on dist-test-slave-q9h9"
server_key: "208cf3f9a01a22619c5d225ddfd8b367"
server_key_iv: "39a449fb3a7141e865b6c684241b7eb0"
server_key_version: "encryptionkey@0"
I20260810 14:36:56.624217 10467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "8ff7ec36375049d38bfc6db829d5a9a8"
format_stamp: "Formatted at 2026-08-10 14:36:56 on dist-test-slave-q9h9"
server_key: "208cf3f9a01a22619c5d225ddfd8b367"
server_key_iv: "39a449fb3a7141e865b6c684241b7eb0"
server_key_version: "encryptionkey@0"
I20260810 14:36:56.629417 10467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 14:36:56.632833 10500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:56.634097 10467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:36:56.634253 10467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "8ff7ec36375049d38bfc6db829d5a9a8"
format_stamp: "Formatted at 2026-08-10 14:36:56 on dist-test-slave-q9h9"
server_key: "208cf3f9a01a22619c5d225ddfd8b367"
server_key_iv: "39a449fb3a7141e865b6c684241b7eb0"
server_key_version: "encryptionkey@0"
I20260810 14:36:56.634428 10467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:56.647990 10467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:56.651655 10467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:56.651896 10467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:56.669651 10467 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:39455
I20260810 14:36:56.669673 10587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:39455 every 8 connection(s)
I20260810 14:36:56.670981 10467 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:56.674606 10588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:56.675746 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 10467
I20260810 14:36:56.675899 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:56.676311 6331 external_mini_cluster.cc:1428] Setting key 0aa6d9d38a30084bb6770877f5f2994d
I20260810 14:36:56.687994 10588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8: Bootstrap starting.
Aug 10 14:36:56 dist-test-slave-q9h9 krb5kdc[10441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372616, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:36:56.690814 10588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:56.692106 10588 log.cc:824] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:56.695261 10588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8: No bootstrap required, opened a new log
I20260810 14:36:56.699364 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:56.678041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52459 (local address 127.6.46.254:39455)
0810 14:36:56.678826 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:56.678840 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:56.678895 (+ 55us) server_negotiation.cc:407] Connection header received
0810 14:36:56.680721 (+ 1826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:56.680750 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:56.681242 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:56.681662 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:56.682979 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:56.684892 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:56.686117 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:56.687108 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:56.687312 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:56.691223 (+ 3911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:56.691234 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:56.691257 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:56.691303 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:56.695057 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:56.695792 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:56.695804 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:56.695812 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:56.696007 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:56.696417 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:56.696421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:56.696424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:56.697081 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:56.697427 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:56.697748 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 14:36:56.698104 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":314,"threads_started":1}
I20260810 14:36:56.703017 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:56.704540 10588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff7ec36375049d38bfc6db829d5a9a8" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39455 } }
I20260810 14:36:56.704855 10588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:56.704947 10588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ff7ec36375049d38bfc6db829d5a9a8, State: Initialized, Role: FOLLOWER
I20260810 14:36:56.705533 10588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff7ec36375049d38bfc6db829d5a9a8" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39455 } }
I20260810 14:36:56.705806 10588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:56.705953 10588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:56.706180 10588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:56.708635 10588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff7ec36375049d38bfc6db829d5a9a8" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39455 } }
I20260810 14:36:56.709079 10588 leader_election.cc:302] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ff7ec36375049d38bfc6db829d5a9a8; no voters:
I20260810 14:36:56.709692 10588 leader_election.cc:288] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:56.709790 10602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:56.711758 10588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:56.716639 10602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [term 1 LEADER]: Becoming Leader. State: Replica: 8ff7ec36375049d38bfc6db829d5a9a8, State: Running, Role: LEADER
I20260810 14:36:56.717293 10602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff7ec36375049d38bfc6db829d5a9a8" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39455 } }
I20260810 14:36:56.720817 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:56.707259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42297 (local address 127.6.46.254:39455)
0810 14:36:56.707410 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:56.707415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:56.707845 (+ 430us) server_negotiation.cc:407] Connection header received
0810 14:36:56.707990 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:56.707996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:56.708057 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:56.708132 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:56.709478 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:56.710640 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:56.711815 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:56.712261 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:56.712369 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:56.714871 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:56.714876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:56.714879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:36:56.714907 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:56.717734 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:56.718524 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:56.718530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:56.718534 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:56.718667 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:36:56.719099 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:36:56.719104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:36:56.719107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:36:56.719425 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:56.719672 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:56.720427 (+ 755us) server_negotiation.cc:299] Negotiation successful
0810 14:36:56.720564 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:36:56.727228 10605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ff7ec36375049d38bfc6db829d5a9a8. Latest consensus state: current_term: 1 leader_uuid: "8ff7ec36375049d38bfc6db829d5a9a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff7ec36375049d38bfc6db829d5a9a8" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39455 } } }
I20260810 14:36:56.727643 10605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:56.728199 10604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ff7ec36375049d38bfc6db829d5a9a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ff7ec36375049d38bfc6db829d5a9a8" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39455 } } }
I20260810 14:36:56.728493 10604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:56.728892 10616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:56.733748 10616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:56.741915 10616 catalog_manager.cc:1347] Generated new cluster ID: dbee4573f8c842afb33c61b8cbff8256
I20260810 14:36:56.742048 10616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:56.770942 10616 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:56.772812 10616 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:56.783700 10616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ff7ec36375049d38bfc6db829d5a9a8: Generated new TSK 0
I20260810 14:36:56.784641 10616 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:56.803720 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10467
I20260810 14:36:56.805049 10579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:36:56.896919 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 10467
W20260810 14:36:56.920318 10439 connection.cc:535] client connection to 127.6.46.254:39455 recv error: Network error: failed to read from TLS socket (remote: 127.6.46.254:39455): Connection reset by peer (error 104)
2026-08-10T14:36:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2120 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:57 dist-test-slave-q9h9 krb5kdc[10632](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:57 dist-test-slave-q9h9 krb5kdc[10632](info): set up 2 sockets
Aug 10 14:36:57 dist-test-slave-q9h9 krb5kdc[10632](info): commencing operation
krb5kdc: starting...
W20260810 14:36:58.123955 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.185s user 0.001s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:36:58 dist-test-slave-q9h9 krb5kdc[10632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372618, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:36:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:36:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:36:58.372749 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:39329
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:35379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:39329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:36:58.703392 10657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:36:58.703663 10657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:36:58.703805 10657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:36:58.710961 10657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:36:58.711071 10657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:36:58.711159 10657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:36:58.711251 10657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:36:58.711314 10657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:36:58.720741 10657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:35379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:39329
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:39329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:36:58.721873 10657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:36:58.723167 10657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:36:58.734584 10671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:58.734535 10675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:36:58.734637 10670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:36:58.735605 10657 server_base.cc:1034] running on GCE node
I20260810 14:36:58.736285 10657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:36:58.737517 10657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:36:58.738729 10657 hybrid_clock.cc:648] HybridClock initialized: now 1786372618738682 us; error 41 us; skew 500 ppm
I20260810 14:36:58.739048 10657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:36:58 dist-test-slave-q9h9 krb5kdc[10632](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372618, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:36:58.746393 10657 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:36:58.747746 10657 webserver.cc:460] Webserver started at http://127.6.46.254:39215/ using document root <none> and password file <none>
I20260810 14:36:58.748358 10657 fs_manager.cc:359] Metadata directory not provided
I20260810 14:36:58.748474 10657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:36:58.748764 10657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:36:58.751613 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "6f87d03c4db946e49f9cc539c8afa5da"
format_stamp: "Formatted at 2026-08-10 14:36:58 on dist-test-slave-q9h9"
server_key: "37eaaeeddec6605a67ded0a2a997d352"
server_key_iv: "28c07c49523e16e5acd91feae7f6dd66"
server_key_version: "encryptionkey@0"
I20260810 14:36:58.752370 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "6f87d03c4db946e49f9cc539c8afa5da"
format_stamp: "Formatted at 2026-08-10 14:36:58 on dist-test-slave-q9h9"
server_key: "37eaaeeddec6605a67ded0a2a997d352"
server_key_iv: "28c07c49523e16e5acd91feae7f6dd66"
server_key_version: "encryptionkey@0"
I20260810 14:36:58.757232 10657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260810 14:36:58.760521 10691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:36:58.761709 10657 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:36:58.761863 10657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "6f87d03c4db946e49f9cc539c8afa5da"
format_stamp: "Formatted at 2026-08-10 14:36:58 on dist-test-slave-q9h9"
server_key: "37eaaeeddec6605a67ded0a2a997d352"
server_key_iv: "28c07c49523e16e5acd91feae7f6dd66"
server_key_version: "encryptionkey@0"
I20260810 14:36:58.762048 10657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:36:58.775300 10657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:36:58.779155 10657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:36:58.779397 10657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:36:58.796007 10657 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:39329
I20260810 14:36:58.796041 10780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:39329 every 8 connection(s)
I20260810 14:36:58.797240 10657 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:36:58.800642 10781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:36:58.803227 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 10657
I20260810 14:36:58.803401 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:36:58.803925 6331 external_mini_cluster.cc:1428] Setting key 1dc084c7f4ec4a704df4fa8883bdf978
I20260810 14:36:58.812352 10781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da: Bootstrap starting.
I20260810 14:36:58.815842 10781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da: Neither blocks nor log segments found. Creating new log.
I20260810 14:36:58.816660 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:58.805572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37107 (local address 127.6.46.254:39329)
0810 14:36:58.806424 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:58.806435 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:58.806473 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:36:58.808059 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:58.808079 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:58.808483 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:58.808820 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:58.809392 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:58.810748 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:58.812518 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:58.813154 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:58.813286 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:58.814552 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:58.814559 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:58.814571 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:58.815084 (+ 513us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:58.815152 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:58.815180 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:58.815294 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:58.815546 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 14:36:58.815720 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":362,"threads_started":1}
I20260810 14:36:58.816958 10781 log.cc:824] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da: Log is configured to *not* fsync() on all Append() calls
I20260810 14:36:58.819779 10781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da: No bootstrap required, opened a new log
I20260810 14:36:58.820178 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:36:58.826483 10781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f87d03c4db946e49f9cc539c8afa5da" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39329 } }
I20260810 14:36:58.826800 10781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:36:58.826907 10781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f87d03c4db946e49f9cc539c8afa5da, State: Initialized, Role: FOLLOWER
I20260810 14:36:58.827491 10781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f87d03c4db946e49f9cc539c8afa5da" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39329 } }
I20260810 14:36:58.827762 10781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:36:58.827891 10781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:36:58.828045 10781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:36:58.830827 10781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f87d03c4db946e49f9cc539c8afa5da" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39329 } }
I20260810 14:36:58.831365 10781 leader_election.cc:302] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f87d03c4db946e49f9cc539c8afa5da; no voters:
I20260810 14:36:58.832011 10781 leader_election.cc:288] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:36:58.832093 10793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:36:58.834089 10781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:36:58.834339 10793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [term 1 LEADER]: Becoming Leader. State: Replica: 6f87d03c4db946e49f9cc539c8afa5da, State: Running, Role: LEADER
I20260810 14:36:58.834823 10793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f87d03c4db946e49f9cc539c8afa5da" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39329 } }
I20260810 14:36:58.835803 10784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:36:58.828487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45597 (local address 127.6.46.254:39329)
0810 14:36:58.828666 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 14:36:58.828675 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:36:58.829181 (+ 506us) server_negotiation.cc:407] Connection header received
0810 14:36:58.829355 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:36:58.829358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:36:58.829417 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:36:58.829490 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:36:58.830149 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:58.830859 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:58.832683 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:36:58.833351 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:36:58.833447 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:36:58.834910 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:36:58.834916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:36:58.834921 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:36:58.834946 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:36:58.835002 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:36:58.835023 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:36:58.835137 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 14:36:58.835413 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 14:36:58.835466 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 14:36:58.837440 10799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f87d03c4db946e49f9cc539c8afa5da. Latest consensus state: current_term: 1 leader_uuid: "6f87d03c4db946e49f9cc539c8afa5da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f87d03c4db946e49f9cc539c8afa5da" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39329 } } }
I20260810 14:36:58.837514 10794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f87d03c4db946e49f9cc539c8afa5da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f87d03c4db946e49f9cc539c8afa5da" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 39329 } } }
I20260810 14:36:58.837944 10799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:58.837986 10794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da [sys.catalog]: This master's current role is: LEADER
I20260810 14:36:58.847083 10808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:36:58.851006 10808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:36:58.859241 10808 catalog_manager.cc:1347] Generated new cluster ID: 7fea486b867c48498e2345f5ff503bf3
I20260810 14:36:58.859328 10808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:36:58.876190 10808 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:36:58.877383 10808 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:36:58.883821 10808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f87d03c4db946e49f9cc539c8afa5da: Generated new TSK 0
I20260810 14:36:58.884529 10808 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:36:58.898803 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10657
W20260810 14:36:58.899839 10772 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37107
W20260810 14:36:58.900521 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10657: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:36:58.900686 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 10657
W20260810 14:36:58.926869 10629 connection.cc:535] client connection to 127.6.46.254:39329 recv error: Network error: failed to read from TLS socket (remote: 127.6.46.254:39329): Connection reset by peer (error 104)
2026-08-10T14:36:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2006 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:36:59 dist-test-slave-q9h9 krb5kdc[10838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:36:59 dist-test-slave-q9h9 krb5kdc[10838](info): set up 2 sockets
Aug 10 14:36:59 dist-test-slave-q9h9 krb5kdc[10838](info): commencing operation
krb5kdc: starting...
W20260810 14:37:00.144088 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.003s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:37:00 dist-test-slave-q9h9 krb5kdc[10838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:37:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:37:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:37:00.405838 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:33775
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:33775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:37:00.726821 10855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:00.727092 10855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:00.727226 10855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:37:00.734102 10855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:37:00.734212 10855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:00.734285 10855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:00.734376 10855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:37:00.734438 10855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:37:00.743736 10855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:33775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:33775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:37:00.744869 10855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:00.746153 10855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:00.757053 10863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:00.757119 10866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:00.757134 10864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:00.758221 10855 server_base.cc:1034] running on GCE node
I20260810 14:37:00.758948 10855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:00.760201 10855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:00.761406 10855 hybrid_clock.cc:648] HybridClock initialized: now 1786372620761364 us; error 31 us; skew 500 ppm
I20260810 14:37:00.761734 10855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:37:00 dist-test-slave-q9h9 krb5kdc[10838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372620, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:00.764671 10855 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:37:00.766039 10855 webserver.cc:460] Webserver started at http://127.6.46.254:33117/ using document root <none> and password file <none>
I20260810 14:37:00.766677 10855 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:00.766804 10855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:00.767089 10855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:37:00.769860 10855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "ea1e19a858e04901bd729d7339f32a83"
format_stamp: "Formatted at 2026-08-10 14:37:00 on dist-test-slave-q9h9"
server_key: "70cc7729836278fb62083f948ec92127"
server_key_iv: "2de9a57200c18368cd81977cc2f9538c"
server_key_version: "encryptionkey@0"
I20260810 14:37:00.770635 10855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "ea1e19a858e04901bd729d7339f32a83"
format_stamp: "Formatted at 2026-08-10 14:37:00 on dist-test-slave-q9h9"
server_key: "70cc7729836278fb62083f948ec92127"
server_key_iv: "2de9a57200c18368cd81977cc2f9538c"
server_key_version: "encryptionkey@0"
I20260810 14:37:00.775369 10855 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260810 14:37:00.778558 10878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:00.779709 10855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:37:00.779860 10855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "ea1e19a858e04901bd729d7339f32a83"
format_stamp: "Formatted at 2026-08-10 14:37:00 on dist-test-slave-q9h9"
server_key: "70cc7729836278fb62083f948ec92127"
server_key_iv: "2de9a57200c18368cd81977cc2f9538c"
server_key_version: "encryptionkey@0"
I20260810 14:37:00.780026 10855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:00.785308 10855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:00.789139 10855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:00.789404 10855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:00.806500 10855 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:33775
I20260810 14:37:00.806553 10961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:33775 every 8 connection(s)
I20260810 14:37:00.807747 10855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:37:00.811170 10962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:37:00.818002 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 10855
I20260810 14:37:00.818203 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:37:00.818616 6331 external_mini_cluster.cc:1428] Setting key 5ae65d03a94852d1482215bea4e30b0d
I20260810 14:37:00.820979 10962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83: Bootstrap starting.
I20260810 14:37:00.824141 10962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:00.825291 10962 log.cc:824] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:00.825816 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:00.820160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42567 (local address 127.6.46.254:33775)
0810 14:37:00.820909 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:00.820920 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:00.820958 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:37:00.822701 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:00.822723 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:00.823023 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:00.823314 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:00.824332 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:00.824338 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:00.824350 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:37:00.824365 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:00.824428 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:37:00.824447 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:00.824611 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:00.824748 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 14:37:00.824911 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":322,"threads_started":1}
I20260810 14:37:00.827844 10962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83: No bootstrap required, opened a new log
I20260810 14:37:00.829079 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:37:00.836297 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:00.833949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60209 (local address 127.6.46.254:33775)
0810 14:37:00.834143 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:00.834149 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:00.834614 (+ 465us) server_negotiation.cc:407] Connection header received
0810 14:37:00.834771 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:00.834774 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:00.834814 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:00.834898 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:00.835718 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:00.835724 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:00.835729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:37:00.835749 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:00.835783 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:37:00.835796 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:00.835906 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:00.835998 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 14:37:00.836052 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 14:37:00.836762 10962 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1e19a858e04901bd729d7339f32a83" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33775 } }
I20260810 14:37:00.837103 10962 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:37:00.837198 10962 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1e19a858e04901bd729d7339f32a83, State: Initialized, Role: FOLLOWER
I20260810 14:37:00.837734 10962 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1e19a858e04901bd729d7339f32a83" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33775 } }
I20260810 14:37:00.837944 10962 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:37:00.838104 10962 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:37:00.838240 10962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:00.840417 10962 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1e19a858e04901bd729d7339f32a83" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33775 } }
I20260810 14:37:00.840819 10962 leader_election.cc:302] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea1e19a858e04901bd729d7339f32a83; no voters:
I20260810 14:37:00.841286 10962 leader_election.cc:288] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:37:00.841405 10977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:37:00.842032 10977 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [term 1 LEADER]: Becoming Leader. State: Replica: ea1e19a858e04901bd729d7339f32a83, State: Running, Role: LEADER
I20260810 14:37:00.842552 10977 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1e19a858e04901bd729d7339f32a83" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33775 } }
I20260810 14:37:00.843116 10962 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:37:00.845379 10979 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea1e19a858e04901bd729d7339f32a83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1e19a858e04901bd729d7339f32a83" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33775 } } }
I20260810 14:37:00.845803 10979 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:00.846236 10982 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea1e19a858e04901bd729d7339f32a83. Latest consensus state: current_term: 1 leader_uuid: "ea1e19a858e04901bd729d7339f32a83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1e19a858e04901bd729d7339f32a83" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 33775 } } }
I20260810 14:37:00.846549 10982 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83 [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:00.851483 10991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:37:00.855837 10991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:37:00.863353 10991 catalog_manager.cc:1347] Generated new cluster ID: 5c292172e09a4707a80e1a23b8e4bfd4
I20260810 14:37:00.863446 10991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:37:00.900440 10991 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:37:00.900815 10991 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:37:00.913790 10991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea1e19a858e04901bd729d7339f32a83: Generated new TSK 0
I20260810 14:37:00.914518 10991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:37:00.919539 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10855
W20260810 14:37:00.920578 10957 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42567
W20260810 14:37:00.921236 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 10855: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:37:00.921388 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 10855
2026-08-10T14:37:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2018 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:37:01 dist-test-slave-q9h9 krb5kdc[11021](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:37:01 dist-test-slave-q9h9 krb5kdc[11021](info): set up 2 sockets
Aug 10 14:37:01 dist-test-slave-q9h9 krb5kdc[11021](info): commencing operation
krb5kdc: starting...
W20260810 14:37:02.163270 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.199s user 0.004s sys 0.035s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:37:02 dist-test-slave-q9h9 krb5kdc[11021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:37:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:37:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:37:02.439208 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43701
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:43701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:37:02.762699 11041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:02.762949 11041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:02.763078 11041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:37:02.769959 11041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:37:02.770077 11041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:02.770165 11041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:02.770246 11041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:37:02.770303 11041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:37:02.779570 11041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:43701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:37:02.780720 11041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:02.781980 11041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:02.793021 11053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:02.793108 11051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:02.793025 11058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:02.793843 11041 server_base.cc:1034] running on GCE node
I20260810 14:37:02.794580 11041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:02.795913 11041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:02.797125 11041 hybrid_clock.cc:648] HybridClock initialized: now 1786372622797075 us; error 40 us; skew 500 ppm
I20260810 14:37:02.797454 11041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:37:02 dist-test-slave-q9h9 krb5kdc[11021](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372622, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:02.800426 11041 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:37:02.801764 11041 webserver.cc:460] Webserver started at http://127.6.46.254:44681/ using document root <none> and password file <none>
I20260810 14:37:02.802435 11041 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:02.802559 11041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:02.802845 11041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:37:02.805717 11041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "cd326954b563448bbaca50a7689a9e4e"
format_stamp: "Formatted at 2026-08-10 14:37:02 on dist-test-slave-q9h9"
server_key: "e1005c6974359f0fedb459790bd6e3de"
server_key_iv: "0261506bfc637fc7f250206c01c77f22"
server_key_version: "encryptionkey@0"
I20260810 14:37:02.806457 11041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "cd326954b563448bbaca50a7689a9e4e"
format_stamp: "Formatted at 2026-08-10 14:37:02 on dist-test-slave-q9h9"
server_key: "e1005c6974359f0fedb459790bd6e3de"
server_key_iv: "0261506bfc637fc7f250206c01c77f22"
server_key_version: "encryptionkey@0"
I20260810 14:37:02.811296 11041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 14:37:02.814590 11069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:02.815832 11041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 14:37:02.815995 11041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "cd326954b563448bbaca50a7689a9e4e"
format_stamp: "Formatted at 2026-08-10 14:37:02 on dist-test-slave-q9h9"
server_key: "e1005c6974359f0fedb459790bd6e3de"
server_key_iv: "0261506bfc637fc7f250206c01c77f22"
server_key_version: "encryptionkey@0"
I20260810 14:37:02.816183 11041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:02.839481 11041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:02.843634 11041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:02.843911 11041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:02.861204 11041 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:43701
I20260810 14:37:02.861227 11162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:43701 every 8 connection(s)
I20260810 14:37:02.862553 11041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:37:02.864283 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 11041
I20260810 14:37:02.864439 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:37:02.864871 6331 external_mini_cluster.cc:1428] Setting key cb2a76435e1fb525c79e735321fcc9f4
I20260810 14:37:02.866551 11164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 14:37:02 dist-test-slave-q9h9 krb5kdc[11021](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372622, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:37:02.879653 11164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e: Bootstrap starting.
I20260810 14:37:02.883519 11164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:02.884754 11164 log.cc:824] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:02.887491 11164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e: No bootstrap required, opened a new log
I20260810 14:37:02.888263 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:02.866610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56965 (local address 127.6.46.254:43701)
0810 14:37:02.867408 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:02.867423 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:02.867481 (+ 58us) server_negotiation.cc:407] Connection header received
0810 14:37:02.869578 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:02.869607 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:02.870135 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:02.870558 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:02.872061 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:02.874057 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:02.875289 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:02.876279 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:02.876464 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:02.880486 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:02.880496 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:02.880513 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:02.880557 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:02.884295 (+ 3738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:02.884987 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:02.884994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:02.885000 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:02.885166 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:02.885538 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:02.885542 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:02.885544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:02.886146 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:02.886425 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:02.886902 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 14:37:02.887282 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":313,"threads_started":1}
I20260810 14:37:02.891783 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:37:02.895051 11164 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd326954b563448bbaca50a7689a9e4e" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43701 } }
I20260810 14:37:02.895380 11164 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:37:02.895488 11164 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd326954b563448bbaca50a7689a9e4e, State: Initialized, Role: FOLLOWER
I20260810 14:37:02.896133 11164 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd326954b563448bbaca50a7689a9e4e" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43701 } }
I20260810 14:37:02.896389 11164 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:37:02.896536 11164 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:37:02.896735 11164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:02.899330 11164 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd326954b563448bbaca50a7689a9e4e" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43701 } }
I20260810 14:37:02.899788 11164 leader_election.cc:302] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd326954b563448bbaca50a7689a9e4e; no voters:
I20260810 14:37:02.900324 11164 leader_election.cc:288] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:37:02.900470 11184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:37:02.902518 11164 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:37:02.902632 11184 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [term 1 LEADER]: Becoming Leader. State: Replica: cd326954b563448bbaca50a7689a9e4e, State: Running, Role: LEADER
I20260810 14:37:02.903198 11184 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd326954b563448bbaca50a7689a9e4e" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43701 } }
I20260810 14:37:02.906172 11188 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd326954b563448bbaca50a7689a9e4e. Latest consensus state: current_term: 1 leader_uuid: "cd326954b563448bbaca50a7689a9e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd326954b563448bbaca50a7689a9e4e" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43701 } } }
I20260810 14:37:02.906548 11188 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:02.910755 11187 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd326954b563448bbaca50a7689a9e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd326954b563448bbaca50a7689a9e4e" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43701 } } }
I20260810 14:37:02.911139 11187 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:02.912644 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:02.895780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50755 (local address 127.6.46.254:43701)
0810 14:37:02.895937 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:02.895943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:02.896459 (+ 516us) server_negotiation.cc:407] Connection header received
0810 14:37:02.896565 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:02.896570 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:02.896644 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:02.896758 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:02.898092 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:02.899137 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:02.900552 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:02.901267 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:02.901384 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:02.904575 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:02.904581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:02.904585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:02.904625 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:02.908479 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:02.909373 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:02.909379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:02.909383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:02.909570 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:02.910039 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:02.910045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:02.910049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:02.910455 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:02.911865 (+ 1410us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:02.912052 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 14:37:02.912287 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 14:37:02.912734 11203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:37:02.917939 11203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:37:02.926590 11203 catalog_manager.cc:1347] Generated new cluster ID: 9f64930e782f4af891126c7ed640fab0
I20260810 14:37:02.926703 11203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:37:02.944528 11203 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:37:02.945730 11203 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:37:02.956521 11203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd326954b563448bbaca50a7689a9e4e: Generated new TSK 0
I20260810 14:37:02.957474 11203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:37:02.965577 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11041
I20260810 14:37:02.966671 11157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:37:03.053597 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 11041
2026-08-10T14:37:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2127 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:37:03 dist-test-slave-q9h9 krb5kdc[11226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:37:03 dist-test-slave-q9h9 krb5kdc[11226](info): set up 2 sockets
Aug 10 14:37:03 dist-test-slave-q9h9 krb5kdc[11226](info): commencing operation
krb5kdc: starting...
W20260810 14:37:04.292615 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.201s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:37:04 dist-test-slave-q9h9 krb5kdc[11226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372624, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:37:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:37:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:37:04.583158 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:35017
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:46081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:35017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:37:04.909026 11244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:04.909305 11244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:04.909417 11244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:37:04.916374 11244 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:37:04.916481 11244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:04.916535 11244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:04.916596 11244 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:37:04.916640 11244 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:37:04.926102 11244 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:46081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:35017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:35017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:37:04.927239 11244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:04.928442 11244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:04.939484 11256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:04.939527 11253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:04.939558 11254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:04.940547 11244 server_base.cc:1034] running on GCE node
I20260810 14:37:04.941248 11244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:04.942538 11244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:04.943732 11244 hybrid_clock.cc:648] HybridClock initialized: now 1786372624943687 us; error 36 us; skew 500 ppm
I20260810 14:37:04.944044 11244 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:37:04 dist-test-slave-q9h9 krb5kdc[11226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372624, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:04.955142 11244 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:37:04.956507 11244 webserver.cc:460] Webserver started at http://127.6.46.254:39777/ using document root <none> and password file <none>
I20260810 14:37:04.957149 11244 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:04.957273 11244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:04.957545 11244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:37:04.960466 11244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "9974db3251824e85a7926ebc8bd115a9"
format_stamp: "Formatted at 2026-08-10 14:37:04 on dist-test-slave-q9h9"
server_key: "dde25c0f37a5d3adff4c12118a601463"
server_key_iv: "2d337962f83c10902d0fc09fe932c437"
server_key_version: "encryptionkey@0"
I20260810 14:37:04.961197 11244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "9974db3251824e85a7926ebc8bd115a9"
format_stamp: "Formatted at 2026-08-10 14:37:04 on dist-test-slave-q9h9"
server_key: "dde25c0f37a5d3adff4c12118a601463"
server_key_iv: "2d337962f83c10902d0fc09fe932c437"
server_key_version: "encryptionkey@0"
I20260810 14:37:04.966198 11244 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260810 14:37:04.969509 11268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:04.970781 11244 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 14:37:04.970939 11244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "9974db3251824e85a7926ebc8bd115a9"
format_stamp: "Formatted at 2026-08-10 14:37:04 on dist-test-slave-q9h9"
server_key: "dde25c0f37a5d3adff4c12118a601463"
server_key_iv: "2d337962f83c10902d0fc09fe932c437"
server_key_version: "encryptionkey@0"
I20260810 14:37:04.971108 11244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:04.989864 11244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:04.993755 11244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:04.994059 11244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:05.011112 11244 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:35017
I20260810 14:37:05.011155 11364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:35017 every 8 connection(s)
I20260810 14:37:05.012341 11244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:37:05.015884 11366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:37:05.018637 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 11244
I20260810 14:37:05.018802 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:37:05.019222 6331 external_mini_cluster.cc:1428] Setting key f7c876251d8ff987d566383ba04a3e49
I20260810 14:37:05.028259 11366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9: Bootstrap starting.
I20260810 14:37:05.031819 11366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:05.033105 11366 log.cc:824] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:05.033617 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:05.020935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37851 (local address 127.6.46.254:35017)
0810 14:37:05.021730 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:05.021744 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:05.021791 (+ 47us) server_negotiation.cc:407] Connection header received
0810 14:37:05.023984 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:05.024011 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:05.024521 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:05.024935 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:05.025426 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:05.027446 (+ 2020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:05.028763 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:05.029744 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:05.029944 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:05.030969 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:05.030979 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:05.030996 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:37:05.031649 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:05.031740 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:37:05.031773 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:05.031894 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:05.032073 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 14:37:05.032362 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":327,"threads_started":1}
I20260810 14:37:05.036145 11366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9: No bootstrap required, opened a new log
I20260810 14:37:05.037329 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:37:05.044153 11366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9974db3251824e85a7926ebc8bd115a9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35017 } }
I20260810 14:37:05.044478 11366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:37:05.044584 11366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9974db3251824e85a7926ebc8bd115a9, State: Initialized, Role: FOLLOWER
I20260810 14:37:05.045125 11366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9974db3251824e85a7926ebc8bd115a9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35017 } }
I20260810 14:37:05.045360 11366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:37:05.045507 11366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:37:05.045674 11366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:05.047438 11371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:05.041772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60335 (local address 127.6.46.254:35017)
0810 14:37:05.041907 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:05.041916 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:05.042370 (+ 454us) server_negotiation.cc:407] Connection header received
0810 14:37:05.042541 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:05.042548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:05.042609 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:05.042692 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:05.043164 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:05.043832 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:05.045187 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:05.045695 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:05.045801 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:05.046784 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:05.046792 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:05.046795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:37:05.046811 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:05.046846 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:37:05.046860 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:05.046946 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:05.047073 (+ 127us) server_negotiation.cc:299] Negotiation successful
0810 14:37:05.047159 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 14:37:05.048332 11366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9974db3251824e85a7926ebc8bd115a9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35017 } }
I20260810 14:37:05.048874 11366 leader_election.cc:302] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9974db3251824e85a7926ebc8bd115a9; no voters:
I20260810 14:37:05.049401 11366 leader_election.cc:288] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:37:05.049511 11383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:37:05.050179 11383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [term 1 LEADER]: Becoming Leader. State: Replica: 9974db3251824e85a7926ebc8bd115a9, State: Running, Role: LEADER
I20260810 14:37:05.050693 11383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9974db3251824e85a7926ebc8bd115a9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35017 } }
I20260810 14:37:05.051373 11366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:37:05.053390 11384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9974db3251824e85a7926ebc8bd115a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9974db3251824e85a7926ebc8bd115a9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35017 } } }
I20260810 14:37:05.053340 11385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9974db3251824e85a7926ebc8bd115a9. Latest consensus state: current_term: 1 leader_uuid: "9974db3251824e85a7926ebc8bd115a9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9974db3251824e85a7926ebc8bd115a9" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 35017 } } }
I20260810 14:37:05.053716 11384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:05.053742 11385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9 [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:05.054513 11395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:37:05.062304 11395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:37:05.070748 11395 catalog_manager.cc:1347] Generated new cluster ID: 91c658d6050540cf886a627064f68c99
I20260810 14:37:05.070842 11395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:37:05.085958 11395 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:37:05.087321 11395 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:37:05.095142 11395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9974db3251824e85a7926ebc8bd115a9: Generated new TSK 0
I20260810 14:37:05.095876 11395 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:37:05.102941 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11244
W20260810 14:37:05.103922 11359 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37851
W20260810 14:37:05.104534 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11244: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:37:05.104650 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 11244
2026-08-10T14:37:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2054 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:37:05 dist-test-slave-q9h9 krb5kdc[11419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:37:05 dist-test-slave-q9h9 krb5kdc[11419](info): set up 2 sockets
Aug 10 14:37:05 dist-test-slave-q9h9 krb5kdc[11419](info): commencing operation
krb5kdc: starting...
W20260810 14:37:06.342839 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.197s user 0.000s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:37:06 dist-test-slave-q9h9 krb5kdc[11419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372626, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:37:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:37:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:37:06.640277 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43833
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:45717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:43833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:37:06.965951 11437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:06.966231 11437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:06.966338 11437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:37:06.973135 11437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:37:06.973227 11437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:06.973287 11437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:06.973353 11437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:37:06.973413 11437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:37:06.982699 11437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:45717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:43833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:43833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:37:06.983771 11437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:06.984942 11437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:06.995842 11445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:06.995779 11451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:06.995808 11446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:06.996516 11437 server_base.cc:1034] running on GCE node
I20260810 14:37:06.997259 11437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:06.998526 11437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:06.999750 11437 hybrid_clock.cc:648] HybridClock initialized: now 1786372626999698 us; error 37 us; skew 500 ppm
I20260810 14:37:07.000087 11437 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:37:07 dist-test-slave-q9h9 krb5kdc[11419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372627, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:07.003077 11437 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:37:07.004472 11437 webserver.cc:460] Webserver started at http://127.6.46.254:36497/ using document root <none> and password file <none>
I20260810 14:37:07.005081 11437 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:07.005206 11437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:07.005477 11437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:37:07.008316 11437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "0d927753ab30475bae7bfa8196d418a8"
format_stamp: "Formatted at 2026-08-10 14:37:07 on dist-test-slave-q9h9"
server_key: "d5c69ecc2d1640dc4ad9f4e98ab041e4"
server_key_iv: "12f2fa5a186a82eecb835ff85ef60eb8"
server_key_version: "encryptionkey@0"
I20260810 14:37:07.009035 11437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "0d927753ab30475bae7bfa8196d418a8"
format_stamp: "Formatted at 2026-08-10 14:37:07 on dist-test-slave-q9h9"
server_key: "d5c69ecc2d1640dc4ad9f4e98ab041e4"
server_key_iv: "12f2fa5a186a82eecb835ff85ef60eb8"
server_key_version: "encryptionkey@0"
I20260810 14:37:07.013842 11437 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260810 14:37:07.017141 11459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:07.018371 11437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 14:37:07.018524 11437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "0d927753ab30475bae7bfa8196d418a8"
format_stamp: "Formatted at 2026-08-10 14:37:07 on dist-test-slave-q9h9"
server_key: "d5c69ecc2d1640dc4ad9f4e98ab041e4"
server_key_iv: "12f2fa5a186a82eecb835ff85ef60eb8"
server_key_version: "encryptionkey@0"
I20260810 14:37:07.018692 11437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:07.023983 11437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:07.027666 11437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:07.027923 11437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:07.044574 11437 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:43833
I20260810 14:37:07.044643 11550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:43833 every 8 connection(s)
I20260810 14:37:07.045819 11437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:37:07.049256 11551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:37:07.055208 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 11437
I20260810 14:37:07.055398 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:37:07.055950 6331 external_mini_cluster.cc:1428] Setting key ffecb4e6073c6af660f3dec3a09a6bce
I20260810 14:37:07.059767 11551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8: Bootstrap starting.
I20260810 14:37:07.063135 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:07.057676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56029 (local address 127.6.46.254:43833)
0810 14:37:07.058475 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:07.058486 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:07.058525 (+ 39us) server_negotiation.cc:407] Connection header received
0810 14:37:07.060143 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:07.060166 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:07.060473 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:07.060770 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:07.061705 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:07.061713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:07.061726 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:37:07.061740 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:07.061795 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:37:07.061813 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:07.061976 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:07.062113 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 14:37:07.062293 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":397,"threads_started":1}
I20260810 14:37:07.063324 11551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:07.064370 11551 log.cc:824] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:07.066735 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:37:07.067445 11551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8: No bootstrap required, opened a new log
I20260810 14:37:07.073851 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:07.071623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42729 (local address 127.6.46.254:43833)
0810 14:37:07.071789 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:07.071794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:07.072172 (+ 378us) server_negotiation.cc:407] Connection header received
0810 14:37:07.072300 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:07.072305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:07.072361 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:07.072462 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:07.073274 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:07.073280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:07.073285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 14:37:07.073303 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:07.073350 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 14:37:07.073369 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:07.073491 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:07.073531 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 14:37:07.073579 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 14:37:07.076535 11551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d927753ab30475bae7bfa8196d418a8" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43833 } }
I20260810 14:37:07.076838 11551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 14:37:07.076944 11551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d927753ab30475bae7bfa8196d418a8, State: Initialized, Role: FOLLOWER
I20260810 14:37:07.077454 11551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d927753ab30475bae7bfa8196d418a8" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43833 } }
I20260810 14:37:07.077687 11551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 14:37:07.077819 11551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 14:37:07.077971 11551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:07.080374 11551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d927753ab30475bae7bfa8196d418a8" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43833 } }
I20260810 14:37:07.080812 11551 leader_election.cc:302] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d927753ab30475bae7bfa8196d418a8; no voters:
I20260810 14:37:07.081316 11551 leader_election.cc:288] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 14:37:07.081432 11574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:37:07.082031 11574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [term 1 LEADER]: Becoming Leader. State: Replica: 0d927753ab30475bae7bfa8196d418a8, State: Running, Role: LEADER
I20260810 14:37:07.082531 11574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d927753ab30475bae7bfa8196d418a8" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43833 } }
I20260810 14:37:07.083163 11551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:37:07.086045 11576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d927753ab30475bae7bfa8196d418a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d927753ab30475bae7bfa8196d418a8" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43833 } } }
I20260810 14:37:07.085168 11578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d927753ab30475bae7bfa8196d418a8. Latest consensus state: current_term: 1 leader_uuid: "0d927753ab30475bae7bfa8196d418a8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d927753ab30475bae7bfa8196d418a8" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 43833 } } }
I20260810 14:37:07.090170 11576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:07.090281 11578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8 [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:07.091454 11592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:37:07.096124 11592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:37:07.104753 11592 catalog_manager.cc:1347] Generated new cluster ID: 12ef9f28ef654d1aa40d9ea806de8920
I20260810 14:37:07.104846 11592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:37:07.137003 11592 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:37:07.137388 11592 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:37:07.146783 11592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d927753ab30475bae7bfa8196d418a8: Generated new TSK 0
I20260810 14:37:07.147639 11592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:37:07.161155 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11437
W20260810 14:37:07.162227 11544 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56029
W20260810 14:37:07.162853 6331 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11437: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 14:37:07.162976 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 11437
2026-08-10T14:37:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2058 ms)
[----------] 18 tests from AuthTokenIssuingTest (37035 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:37:07 dist-test-slave-q9h9 krb5kdc[11609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:37:07 dist-test-slave-q9h9 krb5kdc[11609](info): set up 2 sockets
Aug 10 14:37:07 dist-test-slave-q9h9 krb5kdc[11609](info): commencing operation
krb5kdc: starting...
W20260810 14:37:08.406857 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.202s user 0.000s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:37:08 dist-test-slave-q9h9 krb5kdc[11609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:37:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:37:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:37:08.687104 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:45003
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:45003,127.6.46.253:41321,127.6.46.252:39801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:37:09.012405 11631 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:37:09.012771 11631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:09.012871 11631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:09.012992 11631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:37:09.020265 11631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:37:09.020349 11631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:09.020435 11631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:09.020498 11631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:37:09.020581 11631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:37:09.030009 11631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:45003,127.6.46.253:41321,127.6.46.252:39801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:45003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:37:09.031201 11631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:09.032433 11631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:09.043448 11640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:09.043478 11642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:09.043841 11639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:09.044348 11631 server_base.cc:1034] running on GCE node
I20260810 14:37:09.045024 11631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:09.046283 11631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:09.047490 11631 hybrid_clock.cc:648] HybridClock initialized: now 1786372629047426 us; error 42 us; skew 500 ppm
I20260810 14:37:09.047783 11631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:37:09 dist-test-slave-q9h9 krb5kdc[11609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372629, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:09.050760 11631 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:37:09.052090 11631 webserver.cc:460] Webserver started at http://127.6.46.254:35809/ using document root <none> and password file <none>
I20260810 14:37:09.052681 11631 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:09.052767 11631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:09.053014 11631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:37:09.055806 11631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "b896d54218714c558bd0db8f69cb9c33"
format_stamp: "Formatted at 2026-08-10 14:37:09 on dist-test-slave-q9h9"
server_key: "6f0548014a033af0e064ff81fe835da3"
server_key_iv: "0f81f87d545894ae32df0e776a2e5a23"
server_key_version: "encryptionkey@0"
I20260810 14:37:09.056507 11631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "b896d54218714c558bd0db8f69cb9c33"
format_stamp: "Formatted at 2026-08-10 14:37:09 on dist-test-slave-q9h9"
server_key: "6f0548014a033af0e064ff81fe835da3"
server_key_iv: "0f81f87d545894ae32df0e776a2e5a23"
server_key_version: "encryptionkey@0"
I20260810 14:37:09.061241 11631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 14:37:09.064376 11656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:09.065557 11631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:37:09.065689 11631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "b896d54218714c558bd0db8f69cb9c33"
format_stamp: "Formatted at 2026-08-10 14:37:09 on dist-test-slave-q9h9"
server_key: "6f0548014a033af0e064ff81fe835da3"
server_key_iv: "0f81f87d545894ae32df0e776a2e5a23"
server_key_version: "encryptionkey@0"
I20260810 14:37:09.065850 11631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:09.078948 11631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:09.082810 11631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:09.083006 11631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:09.099627 11631 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:45003
I20260810 14:37:09.099658 11738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:45003 every 8 connection(s)
I20260810 14:37:09.100813 11631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:37:09.104200 11739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:37:09.110208 11739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:37:09.111147 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 11631
I20260810 14:37:09.111372 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:37:09.112186 6331 external_mini_cluster.cc:1428] Setting key 452f622b602910daca4ed5abd4a97789
Aug 10 14:37:09 dist-test-slave-q9h9 krb5kdc[11609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372629, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:37:09.131723 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.113461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:45003 (local address 127.0.0.1:47780)
0810 14:37:09.114686 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.114740 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:09.116386 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:09.117213 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:09.117217 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:09.117748 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:09.118104 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:09.118122 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.120101 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.120105 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:09.121128 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:09.121140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.122214 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.122219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:09.122337 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:09.122888 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:09.122908 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:09.125129 (+ 2221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:09.128272 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:09.128281 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:09.128288 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:09.128661 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:09.129090 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:09.129094 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:09.129096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:09.129241 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:09.130092 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:09.130097 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:09.130270 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 14:37:09.130460 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 14:37:09.130793 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":964,"spinlock_wait_cycles":33152,"thread_start_us":756,"threads_started":1}
I20260810 14:37:09.131758 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.113461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47780 (local address 127.6.46.254:45003)
0810 14:37:09.114319 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:09.114331 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:09.114867 (+ 536us) server_negotiation.cc:407] Connection header received
0810 14:37:09.116722 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:09.116729 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:09.116986 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:09.117178 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:09.118298 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.119936 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.121263 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.122006 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.122195 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:09.125341 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:09.125349 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:09.125359 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:09.125387 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:09.128155 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:09.128783 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:09.128791 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:09.128795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:09.128972 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:09.129351 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:09.129356 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:09.129359 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:09.129946 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:09.130156 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:09.130620 (+ 464us) server_negotiation.cc:299] Negotiation successful
0810 14:37:09.130968 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":635,"thread_start_us":387,"threads_started":1}
I20260810 14:37:09.134476 11739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:37:09.135540 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.135191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41321 (local address 127.0.0.1:59104)
0810 14:37:09.135344 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.135446 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 14:37:09.135835 11661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
I20260810 14:37:09.137260 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.136940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41321 (local address 127.0.0.1:59110)
0810 14:37:09.137063 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.137127 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 14:37:09.137459 11739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:41321: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.6.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.253.keytab.
Entry for principal kudu/127.6.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.253.keytab.
I20260810 14:37:09.174634 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.253:41321
--webserver_interface=127.6.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.253.keytab
--principal=kudu/127.6.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:45003,127.6.46.253:41321,127.6.46.252:39801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:37:09.186713 11739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } attempt: 1
I20260810 14:37:09.187668 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.187269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41321 (local address 127.0.0.1:59114)
0810 14:37:09.187445 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.187527 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 14:37:09.188592 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.188233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41321 (local address 127.0.0.1:59128)
0810 14:37:09.188372 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.188438 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 14:37:09.188766 11739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:41321: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
I20260810 14:37:09.257018 11739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } attempt: 2
I20260810 14:37:09.258250 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.257755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41321 (local address 127.0.0.1:59142)
0810 14:37:09.257949 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.258083 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 14:37:09.259284 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.258892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41321 (local address 127.0.0.1:59154)
0810 14:37:09.259050 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.259143 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 14:37:09.259495 11739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:41321: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
I20260810 14:37:09.350765 11739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } attempt: 3
I20260810 14:37:09.351958 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.351492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41321 (local address 127.0.0.1:59162)
0810 14:37:09.351709 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.351809 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 14:37:09.352999 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.352619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41321 (local address 127.0.0.1:59172)
0810 14:37:09.352766 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.352840 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 14:37:09.353195 11739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:41321: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
W20260810 14:37:09.489460 11754 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:37:09.489768 11754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:09.489862 11754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:09.489974 11754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 14:37:09.496408 11739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } attempt: 4
W20260810 14:37:09.496852 11754 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:37:09.496932 11754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:09.497021 11754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:09.497099 11754 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:37:09.497184 11754 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:37:09.497622 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.497027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41321 (local address 127.0.0.1:59176)
0810 14:37:09.497384 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.497480 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
I20260810 14:37:09.498574 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.498235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41321 (local address 127.0.0.1:59178)
0810 14:37:09.498372 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.498454 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 14:37:09.498777 11739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:41321: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41321: connect: Connection refused (error 111)
I20260810 14:37:09.506649 11754 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:45003,127.6.46.253:41321,127.6.46.252:39801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.253.keytab
--principal=kudu/127.6.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.253:41321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:37:09.507766 11754 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:09.508947 11754 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:09.519618 11761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:09.519531 11765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:09.519702 11763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:09.520485 11754 server_base.cc:1034] running on GCE node
I20260810 14:37:09.521118 11754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:09.522334 11754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:09.523535 11754 hybrid_clock.cc:648] HybridClock initialized: now 1786372629523473 us; error 34 us; skew 500 ppm
I20260810 14:37:09.523806 11754 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:37:09 dist-test-slave-q9h9 krb5kdc[11609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372629, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:09.526698 11754 init.cc:377] Logged in from keytab as kudu/127.6.46.253@KRBTEST.COM (short username kudu)
I20260810 14:37:09.528049 11754 webserver.cc:460] Webserver started at http://127.6.46.253:41499/ using document root <none> and password file <none>
I20260810 14:37:09.528659 11754 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:09.528743 11754 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:09.528990 11754 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:37:09.531669 11754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/data/instance:
uuid: "4d5bdc5f41144ec49157359734c6ed06"
format_stamp: "Formatted at 2026-08-10 14:37:09 on dist-test-slave-q9h9"
server_key: "8bf531a02ac5230616f8e70af7c0fecb"
server_key_iv: "176cb4b1f3fb272c04918f6677962c90"
server_key_version: "encryptionkey@0"
I20260810 14:37:09.532346 11754 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/wal/instance:
uuid: "4d5bdc5f41144ec49157359734c6ed06"
format_stamp: "Formatted at 2026-08-10 14:37:09 on dist-test-slave-q9h9"
server_key: "8bf531a02ac5230616f8e70af7c0fecb"
server_key_iv: "176cb4b1f3fb272c04918f6677962c90"
server_key_version: "encryptionkey@0"
I20260810 14:37:09.536989 11754 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260810 14:37:09.540151 11776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:09.541352 11754 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 14:37:09.541484 11754 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/wal
uuid: "4d5bdc5f41144ec49157359734c6ed06"
format_stamp: "Formatted at 2026-08-10 14:37:09 on dist-test-slave-q9h9"
server_key: "8bf531a02ac5230616f8e70af7c0fecb"
server_key_iv: "176cb4b1f3fb272c04918f6677962c90"
server_key_version: "encryptionkey@0"
I20260810 14:37:09.541656 11754 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:09.557624 11754 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:09.561074 11754 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:09.561278 11754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:09.577832 11754 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.253:41321
I20260810 14:37:09.577864 11852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.253:41321 every 8 connection(s)
I20260810 14:37:09.579023 11754 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/data/info.pb
I20260810 14:37:09.582371 11854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:37:09.588405 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 11754
I20260810 14:37:09.588598 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-1/wal/instance
I20260810 14:37:09.589143 6331 external_mini_cluster.cc:1428] Setting key a1df1b8a00ef092c3cd2cd20ddead4e1
I20260810 14:37:09.588680 11854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:37:09 dist-test-slave-q9h9 krb5kdc[11609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372629, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.253@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:37:09.609680 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.591600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47792 (local address 127.6.46.254:45003)
0810 14:37:09.591803 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:09.591809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:09.592914 (+ 1105us) server_negotiation.cc:407] Connection header received
0810 14:37:09.595120 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:09.595124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:09.595198 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:09.595280 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:09.597029 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.597680 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.599616 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.600013 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.600104 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:09.604054 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:09.604061 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:09.604065 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:09.604100 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:09.606848 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:09.607530 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:09.607534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:09.607537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:09.607671 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:09.608064 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:09.608068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:09.608070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:09.608350 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:09.608484 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:09.609127 (+ 643us) server_negotiation.cc:299] Negotiation successful
0810 14:37:09.609356 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:37:09.610512 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.591981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:45003 (local address 127.0.0.1:47792)
0810 14:37:09.592724 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.592783 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:09.594863 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:09.595513 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:09.595522 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:09.596162 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:09.596883 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:09.596906 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.597832 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.597837 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:09.599487 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:09.599502 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.600217 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.600224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:09.600487 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:09.601146 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:09.601173 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:09.603809 (+ 2636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:09.606969 (+ 3160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:09.606975 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:09.606986 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:09.607400 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:09.607777 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:09.607781 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:09.607783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:09.607946 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:09.608476 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:09.608483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:09.608767 (+ 284us) client_negotiation.cc:769] Sending connection context
0810 14:37:09.608982 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 14:37:09.609428 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":296,"threads_started":1}
I20260810 14:37:09.612212 11854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:37:09 dist-test-slave-q9h9 krb5kdc[11609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372629, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.253@KRBTEST.COM for kudu/127.6.46.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.46.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.252@KRBTEST.COM" created.
I20260810 14:37:09.626556 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.612960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41321 (local address 127.0.0.1:59186)
0810 14:37:09.613117 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.613136 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:09.613257 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:09.613987 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:09.613991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:09.614355 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:09.614480 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:09.614488 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.616252 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.616256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:09.617053 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:09.617063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.618067 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.618071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:09.618205 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:09.618695 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:09.618711 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:09.620612 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:09.624109 (+ 3497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:09.624113 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:09.624115 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:09.624483 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:09.624816 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:09.624819 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:09.624821 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:09.624943 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:09.625497 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:09.625502 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:09.625685 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 14:37:09.625839 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 14:37:09.626136 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 14:37:09.626682 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.612980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.6.46.253:41321)
0810 14:37:09.613553 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:09.613560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:09.613588 (+ 28us) server_negotiation.cc:407] Connection header received
0810 14:37:09.613641 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:09.613658 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:09.613843 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:09.613984 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:09.614661 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.616133 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.617187 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.617914 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.618066 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:09.620823 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:09.620837 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:09.620852 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:09.620887 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:09.623990 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:09.624604 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:09.624610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:09.624612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:09.624739 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:09.625025 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:09.625029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:09.625031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:09.625336 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:09.625513 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:09.626004 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 14:37:09.626259 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":416,"thread_start_us":291,"threads_started":1}
I20260810 14:37:09.629424 11854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:37:09.630517 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.630010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33576)
0810 14:37:09.630235 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.630361 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 14:37:09.630865 11778 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
I20260810 14:37:09.633282 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.632930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33580)
0810 14:37:09.633077 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.633171 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 14:37:09.633481 11854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:39801: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.252.keytab.
Entry for principal kudu/127.6.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.252.keytab.
I20260810 14:37:09.656572 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.252:39801
--webserver_interface=127.6.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.252.keytab
--principal=kudu/127.6.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:39339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:45003,127.6.46.253:41321,127.6.46.252:39801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:37:09.682770 11854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } attempt: 1
I20260810 14:37:09.683859 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.683428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33586)
0810 14:37:09.683600 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.683712 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 14:37:09.684787 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.684446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33600)
0810 14:37:09.684588 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.684682 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 14:37:09.684993 11854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:39801: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
I20260810 14:37:09.753191 11854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } attempt: 2
I20260810 14:37:09.754323 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.753861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33612)
0810 14:37:09.754071 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.754167 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 14:37:09.755306 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.754979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33626)
0810 14:37:09.755121 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.755191 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 14:37:09.755505 11854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:39801: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
I20260810 14:37:09.798074 11739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } attempt: 5
Aug 10 14:37:09 dist-test-slave-q9h9 krb5kdc[11609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372629, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for kudu/127.6.46.253@KRBTEST.COM
I20260810 14:37:09.811414 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.799012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.6.46.253:41321)
0810 14:37:09.799205 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:09.799210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:09.799232 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:37:09.799313 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:09.799316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:09.799378 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:09.799462 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:09.800295 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.800991 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.802156 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.802874 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.803069 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:09.806403 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:09.806408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:09.806411 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:09.806444 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:09.808876 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:09.809672 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:09.809676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:09.809678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:09.809791 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:09.810195 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:09.810198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:09.810200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:09.810460 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:09.810568 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:09.811012 (+ 444us) server_negotiation.cc:299] Negotiation successful
0810 14:37:09.811184 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:37:09.811630 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.798816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41321 (local address 127.0.0.1:59198)
0810 14:37:09.799008 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.799026 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:09.799169 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:09.799527 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:09.799532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:09.799949 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:09.800126 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:09.800135 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.801128 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.801134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:09.801975 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:09.801987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:09.803079 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:09.803083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:09.803253 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:09.803812 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:09.803830 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:09.806221 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:09.809021 (+ 2800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:09.809026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:09.809030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:09.809560 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:09.809890 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:09.809896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:09.809899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:09.810104 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:09.810569 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:09.810574 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:09.810776 (+ 202us) client_negotiation.cc:769] Sending connection context
0810 14:37:09.810922 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 14:37:09.811189 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 14:37:09.812587 11739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:37:09.813565 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.813216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33632)
0810 14:37:09.813358 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.813430 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 14:37:09.814448 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.814157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33648)
0810 14:37:09.814272 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.814356 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 14:37:09.814651 11739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:39801: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
I20260810 14:37:09.846692 11854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } attempt: 3
I20260810 14:37:09.847740 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.847281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33664)
0810 14:37:09.847485 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.847579 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 14:37:09.849081 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.848516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33672)
0810 14:37:09.848778 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.848866 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260810 14:37:09.849398 11854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:39801: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
I20260810 14:37:09.865863 11739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } attempt: 1
I20260810 14:37:09.866884 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.866485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33678)
0810 14:37:09.866648 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.866731 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 14:37:09.867782 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.867503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33692)
0810 14:37:09.867616 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.867683 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 14:37:09.867975 11739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:39801: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
I20260810 14:37:09.936172 11739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } attempt: 2
I20260810 14:37:09.937328 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.936862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33700)
0810 14:37:09.937059 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.937171 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 14:37:09.938288 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.937953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33716)
0810 14:37:09.938097 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.938168 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 14:37:09.938490 11739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:39801: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
W20260810 14:37:09.979939 11866 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:37:09.980249 11866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:09.980347 11866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:09.980456 11866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:37:09.987370 11866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:37:09.987447 11866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:09.987517 11866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:09.987586 11866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:37:09.987672 11866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:37:09.992625 11854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } attempt: 4
I20260810 14:37:09.993734 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.993275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33720)
0810 14:37:09.993490 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.993575 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 14:37:09.994626 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:09.994351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33724)
0810 14:37:09.994462 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:09.994527 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 14:37:09.994830 11854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:39801: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
I20260810 14:37:09.997493 11866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:39339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:45003,127.6.46.253:41321,127.6.46.252:39801
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.252.keytab
--principal=kudu/127.6.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.252:39801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:37:09.998680 11866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:09.999899 11866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:10.010862 11874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:10.010823 11875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:10.010900 11877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:10.011822 11866 server_base.cc:1034] running on GCE node
I20260810 14:37:10.012506 11866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:10.013728 11866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:10.014966 11866 hybrid_clock.cc:648] HybridClock initialized: now 1786372630014898 us; error 33 us; skew 500 ppm
I20260810 14:37:10.015285 11866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:37:10 dist-test-slave-q9h9 krb5kdc[11609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372630, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:10.021850 11866 init.cc:377] Logged in from keytab as kudu/127.6.46.252@KRBTEST.COM (short username kudu)
I20260810 14:37:10.023166 11866 webserver.cc:460] Webserver started at http://127.6.46.252:46211/ using document root <none> and password file <none>
I20260810 14:37:10.023772 11866 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:10.023873 11866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:10.024171 11866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:37:10.026975 11866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/data/instance:
uuid: "79d4f8e2944b4f3b9c5e2f45b985a428"
format_stamp: "Formatted at 2026-08-10 14:37:10 on dist-test-slave-q9h9"
server_key: "b7405339487110be157a4079a62ded2c"
server_key_iv: "8c8dea466c0004a135698198fe152b20"
server_key_version: "encryptionkey@0"
I20260810 14:37:10.027700 11866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/wal/instance:
uuid: "79d4f8e2944b4f3b9c5e2f45b985a428"
format_stamp: "Formatted at 2026-08-10 14:37:10 on dist-test-slave-q9h9"
server_key: "b7405339487110be157a4079a62ded2c"
server_key_iv: "8c8dea466c0004a135698198fe152b20"
server_key_version: "encryptionkey@0"
I20260810 14:37:10.032505 11866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 14:37:10.035682 11889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:10.036831 11866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 14:37:10.036983 11866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/wal
uuid: "79d4f8e2944b4f3b9c5e2f45b985a428"
format_stamp: "Formatted at 2026-08-10 14:37:10 on dist-test-slave-q9h9"
server_key: "b7405339487110be157a4079a62ded2c"
server_key_iv: "8c8dea466c0004a135698198fe152b20"
server_key_version: "encryptionkey@0"
I20260810 14:37:10.037171 11866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:10.044675 11739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } attempt: 3
I20260810 14:37:10.045710 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.045201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33740)
0810 14:37:10.045427 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:10.045522 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 14:37:10.046722 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.046332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33744)
0810 14:37:10.046519 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:10.046596 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260810 14:37:10.046912 11739 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:39801: Network error: Client connection negotiation failed: client connection to 127.6.46.252:39801: connect: Connection refused (error 111)
I20260810 14:37:10.052316 11866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:10.055996 11866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:10.056200 11866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:10.072469 11866 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.252:39801
I20260810 14:37:10.072487 11975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.252:39801 every 8 connection(s)
I20260810 14:37:10.073624 11866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/data/info.pb
I20260810 14:37:10.077013 11977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:37:10.081935 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 11866
I20260810 14:37:10.082118 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786372275129649-6331-0/minicluster-data/master-2/wal/instance
I20260810 14:37:10.082521 6331 external_mini_cluster.cc:1428] Setting key 9d6a7913625b3a943f506a538c07c706
I20260810 14:37:10.083034 11977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:37:10 dist-test-slave-q9h9 krb5kdc[11609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372630, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.252@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
Aug 10 14:37:10 dist-test-slave-q9h9 krb5kdc[11609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.252@KRBTEST.COM
I20260810 14:37:10.101730 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.085884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.6.46.254:45003)
0810 14:37:10.086108 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.086115 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.086721 (+ 606us) server_negotiation.cc:407] Connection header received
0810 14:37:10.087389 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.087392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.087454 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.087523 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.088823 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.089499 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.091146 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.091730 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.091820 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.095105 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:10.095110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:10.095114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:10.095145 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:10.098312 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.099170 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.099175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.099179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.099386 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.099867 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.099872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.099874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.100218 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:10.100350 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.101137 (+ 787us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.101351 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:37:10.102705 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.085822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:45003 (local address 127.0.0.1:47794)
0810 14:37:10.086568 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:10.086618 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:10.087152 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:10.087611 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:10.087618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:10.088325 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:10.088662 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:10.088679 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.089678 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.089684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:10.091010 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:10.091022 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.091943 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.091950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:10.092186 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.092885 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:10.092905 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:10.094909 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:10.098430 (+ 3521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:10.098439 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:10.098446 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:10.099022 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:10.099506 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:10.099511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:10.099515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:10.099737 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:10.100334 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:10.100340 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:10.100709 (+ 369us) client_negotiation.cc:769] Sending connection context
0810 14:37:10.100996 (+ 287us) client_negotiation.cc:240] Negotiation successful
0810 14:37:10.101426 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":344,"threads_started":1}
I20260810 14:37:10.102948 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.084133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.6.46.252:39801)
0810 14:37:10.084891 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.084904 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.084950 (+ 46us) server_negotiation.cc:407] Connection header received
0810 14:37:10.087143 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.087168 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.087442 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.087631 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.088860 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.090222 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.091561 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.092209 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.092338 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.096011 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:10.096023 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:10.096036 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:10.096080 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:10.099119 (+ 3039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.099837 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.099845 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.099848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.100004 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.100422 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.100427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.100430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.100897 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:10.101175 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.102294 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.102518 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":314,"threads_started":1}
I20260810 14:37:10.104401 11977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:37:10 dist-test-slave-q9h9 krb5kdc[11609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372630, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.252@KRBTEST.COM for kudu/127.6.46.253@KRBTEST.COM
Aug 10 14:37:10 dist-test-slave-q9h9 krb5kdc[11609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:37:10.117410 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.105194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41321 (local address 127.0.0.1:59200)
0810 14:37:10.105329 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:10.105346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:10.105441 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:10.105729 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:10.105734 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:10.106196 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:10.106349 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:10.106356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.107279 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.107285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:10.108206 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:10.108221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.109150 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.109157 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:10.109334 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.109826 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:10.109846 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:10.111958 (+ 2112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:10.115161 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:10.115166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:10.115170 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:10.115519 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:10.115911 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:10.115915 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:10.115917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:10.116062 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:10.116588 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:10.116592 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:10.116718 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 14:37:10.116835 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 14:37:10.117086 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 14:37:10.117527 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.105124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59200 (local address 127.6.46.253:41321)
0810 14:37:10.105287 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.105293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.105404 (+ 111us) server_negotiation.cc:407] Connection header received
0810 14:37:10.105540 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.105545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.105614 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.105699 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.106485 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.107141 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.108355 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.109003 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.109126 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.112126 (+ 3000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:10.112131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:10.112134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:10.112169 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:10.115027 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.115625 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.115630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.115633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.115776 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.116193 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.116197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.116200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.116465 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:10.116598 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.116962 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.117173 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:37:10.118372 11977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:37:10.120839 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.106161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47806 (local address 127.6.46.254:45003)
0810 14:37:10.106302 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.106307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.106323 (+ 16us) server_negotiation.cc:407] Connection header received
0810 14:37:10.106368 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.106372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.106430 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.106522 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.108046 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.108883 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.110515 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.110928 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.111051 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.115555 (+ 4504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:10.115560 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:10.115563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:10.115599 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:10.118193 (+ 2594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.118845 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.118849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.118851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.119018 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.119418 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.119422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.119425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.119714 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:10.119912 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.120317 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.120526 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 14:37:10 dist-test-slave-q9h9 krb5kdc[11609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372630, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.252@KRBTEST.COM for kudu/127.6.46.252@KRBTEST.COM
Aug 10 14:37:10 dist-test-slave-q9h9 krb5kdc[11609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372628, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.253@KRBTEST.COM
I20260810 14:37:10.131062 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.119084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.6.46.252:39801)
0810 14:37:10.119232 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.119236 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.119248 (+ 12us) server_negotiation.cc:407] Connection header received
0810 14:37:10.119281 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.119284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.119330 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.119417 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.120082 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.121012 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.122222 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.122658 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.122767 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.125535 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:10.125541 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:10.125545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:10.125579 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:10.128437 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.129124 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.129130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.129134 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.129273 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.129598 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.129603 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.129606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.129932 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:10.130082 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.130490 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.130804 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":70}
I20260810 14:37:10.131071 11982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.118911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33762)
0810 14:37:10.119071 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:10.119089 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:10.119177 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:10.119448 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:10.119451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:10.119783 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:10.119946 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:10.119957 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.121141 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.121145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:10.122075 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:10.122091 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.122788 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.122794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:10.122987 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.123452 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:10.123470 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:10.125356 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:10.128558 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:10.128562 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:10.128565 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:10.129009 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:10.129370 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:10.129373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:10.129375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:10.129494 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:10.130085 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:10.130090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:10.130248 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 14:37:10.130403 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 14:37:10.130698 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 14:37:10.136406 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.122239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.6.46.253:41321)
0810 14:37:10.122384 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.122389 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.122426 (+ 37us) server_negotiation.cc:407] Connection header received
0810 14:37:10.122580 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.122586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.122643 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.122723 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.123956 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.124904 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.126339 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.126732 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.126823 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.131385 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:10.131391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:10.131396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:10.131430 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:10.133905 (+ 2475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.134645 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.134649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.134651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.134794 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.135213 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.135217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.135220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.135479 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:10.135677 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.135987 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.136160 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:37:10.136708 11977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428: Bootstrap starting.
I20260810 14:37:10.140359 11977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:10.140995 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:37:10.141491 11977 log.cc:824] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:10.144070 11977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428: No bootstrap required, opened a new log
I20260810 14:37:10.144040 11697 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b896d54218714c558bd0db8f69cb9c33"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47806
I20260810 14:37:10.150993 11977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } }
I20260810 14:37:10.151280 11977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d4f8e2944b4f3b9c5e2f45b985a428, State: Initialized, Role: FOLLOWER
I20260810 14:37:10.151682 11977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } }
I20260810 14:37:10.152515 11987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } } }
I20260810 14:37:10.152909 11987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:10.153330 11977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 14:37:10.161784 12001 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:37:10.161901 12001 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:37:10.159510 11866 master_runner.cc:186] Error getting master registration for 127.6.46.254:45003: OK, instance_id { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" instance_seqno: 1786372629094857 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:37:10.170068 11866 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:37:10.224118 11739 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } attempt: 4
Aug 10 14:37:10 dist-test-slave-q9h9 krb5kdc[11609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372629, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for kudu/127.6.46.252@KRBTEST.COM
I20260810 14:37:10.240628 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.224891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33766 (local address 127.6.46.252:39801)
0810 14:37:10.225093 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.225098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.225136 (+ 38us) server_negotiation.cc:407] Connection header received
0810 14:37:10.225398 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.225402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.225470 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.225573 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.226563 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.227718 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.229536 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.230265 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.230393 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.234140 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:10.234147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:10.234161 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:10.234200 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:10.237674 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.238458 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.238463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.238467 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.238613 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.238992 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.238997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.239000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.239437 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:10.239619 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.240015 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.240252 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:37:10.240645 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.224823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33766)
0810 14:37:10.225053 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:10.225075 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:10.225231 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:10.225626 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:10.225632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:10.226176 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:10.226399 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:10.226412 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.227877 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.227882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:10.229386 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:10.229403 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.230410 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.230416 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:10.230621 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.231274 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:10.231297 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:10.233942 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:10.237799 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:10.237805 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:10.237809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:10.238331 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:10.238717 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:10.238722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:10.238726 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:10.238880 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:10.239561 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:10.239565 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:10.239743 (+ 178us) client_negotiation.cc:769] Sending connection context
0810 14:37:10.239937 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 14:37:10.240238 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 14:37:10.245666 11739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33: Bootstrap starting.
I20260810 14:37:10.248723 11739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:10.249938 11739 log.cc:824] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:10.252527 11739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33: No bootstrap required, opened a new log
I20260810 14:37:10.259546 11739 raft_consensus.cc:348] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } }
I20260810 14:37:10.259908 11739 raft_consensus.cc:729] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b896d54218714c558bd0db8f69cb9c33, State: Initialized, Role: FOLLOWER
I20260810 14:37:10.260310 11739 consensus_queue.cc:260] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } }
I20260810 14:37:10.261171 12009 sys_catalog.cc:455] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } } }
I20260810 14:37:10.261533 12009 sys_catalog.cc:458] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:10.262228 11739 sys_catalog.cc:564] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:37:10.262426 11697 raft_consensus.cc:482] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 14:37:10.262629 11697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:10.265938 11697 raft_consensus.cc:504] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } }
I20260810 14:37:10.268226 11932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b896d54218714c558bd0db8f69cb9c33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428"
I20260810 14:37:10.268728 11932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:10.271006 11697 leader_election.cc:288] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [CANDIDATE]: Term 1 election: Requested vote from peers 4d5bdc5f41144ec49157359734c6ed06 (127.6.46.253:41321), 79d4f8e2944b4f3b9c5e2f45b985a428 (127.6.46.252:39801)
I20260810 14:37:10.272044 11932 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b896d54218714c558bd0db8f69cb9c33 in term 1.
I20260810 14:37:10.272635 11658 leader_election.cc:302] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79d4f8e2944b4f3b9c5e2f45b985a428, b896d54218714c558bd0db8f69cb9c33; no voters:
I20260810 14:37:10.273448 12009 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:37:10.274924 12009 raft_consensus.cc:686] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [term 1 LEADER]: Becoming Leader. State: Replica: b896d54218714c558bd0db8f69cb9c33, State: Running, Role: LEADER
I20260810 14:37:10.275441 12009 consensus_queue.cc:237] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } }
I20260810 14:37:10.284422 12033 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 14:37:10.284152 12020 sys_catalog.cc:455] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b896d54218714c558bd0db8f69cb9c33. Latest consensus state: current_term: 1 leader_uuid: "b896d54218714c558bd0db8f69cb9c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } } }
I20260810 14:37:10.284607 12020 sys_catalog.cc:458] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:10.285655 12034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:37:10.294003 11854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } attempt: 5
Aug 10 14:37:10 dist-test-slave-q9h9 krb5kdc[11609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372629, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.253@KRBTEST.COM for kudu/127.6.46.252@KRBTEST.COM
I20260810 14:37:10.304952 12034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 14:37:10.307953 11897 tablet.cc:2369] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:37:10.289000 11631 master_runner.cc:186] Error getting master registration for 127.6.46.253:41321: OK, instance_id { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" instance_seqno: 1786372629573167 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:37:10.308665 11631 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:37:10.311715 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.294615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.6.46.252:39801)
0810 14:37:10.295162 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.295169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.295186 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:37:10.295238 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.295243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.295311 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.295397 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.296372 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.297451 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.299222 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.299910 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.300023 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.304812 (+ 4789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:10.304819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:10.304824 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:10.304867 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:10.308440 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.309292 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.309298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.309301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.309469 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.309991 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.309996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.309999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.310378 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:10.310499 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.311120 (+ 621us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.311343 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428}
I20260810 14:37:10.312618 11856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.294555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:39801 (local address 127.0.0.1:33768)
0810 14:37:10.294827 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:10.294847 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:10.294956 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:10.295440 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:10.295444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:10.295988 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:10.296221 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:10.296234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.297614 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.297620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:10.299054 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:10.299070 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.300046 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.300052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:10.300273 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.300973 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:10.300999 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:10.304600 (+ 3601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:10.308600 (+ 4000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:10.308605 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:10.308609 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:10.309146 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:10.309639 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:10.309643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:10.309647 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:10.309855 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:10.310600 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:10.310605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:10.310797 (+ 192us) client_negotiation.cc:769] Sending connection context
0810 14:37:10.311948 (+ 1151us) client_negotiation.cc:240] Negotiation successful
0810 14:37:10.312243 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122}
I20260810 14:37:10.316707 11932 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428 [term 1 FOLLOWER]: Refusing update from remote peer b896d54218714c558bd0db8f69cb9c33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:37:10.317837 12020 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:37:10.318557 11854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06: Bootstrap starting.
I20260810 14:37:10.322875 11854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:10.324324 11854 log.cc:824] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:10.331470 11854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06: No bootstrap required, opened a new log
W20260810 14:37:10.335346 11665 tablet.cc:2369] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:37:10.340579 11987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b896d54218714c558bd0db8f69cb9c33. Latest consensus state: current_term: 1 leader_uuid: "b896d54218714c558bd0db8f69cb9c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } } }
I20260810 14:37:10.340867 11987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:10.342516 11854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } }
I20260810 14:37:10.342893 11854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d5bdc5f41144ec49157359734c6ed06, State: Initialized, Role: FOLLOWER
I20260810 14:37:10.343377 11854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } }
I20260810 14:37:10.347254 12039 mvcc.cc:205] Tried to move back new op lower bound from 7316982293764108288 to 7316982293613203456. Current Snapshot: MvccSnapshot[applied={T|T < 7316982293764108288}]
I20260810 14:37:10.351408 12045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } } }
I20260810 14:37:10.351833 12045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:10.353579 11854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:37:10.353657 11818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b896d54218714c558bd0db8f69cb9c33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4d5bdc5f41144ec49157359734c6ed06"
I20260810 14:37:10.354099 11818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:10.357882 11818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b896d54218714c558bd0db8f69cb9c33 in term 1.
I20260810 14:37:10.358788 11817 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06 [term 1 FOLLOWER]: Refusing update from remote peer b896d54218714c558bd0db8f69cb9c33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:37:10.360148 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.327076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59214 (local address 127.6.46.253:41321)
0810 14:37:10.327245 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.327249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.338589 (+ 11340us) server_negotiation.cc:407] Connection header received
0810 14:37:10.338852 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.338862 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.338937 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.339007 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.341396 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.342579 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.344470 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.345125 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.345238 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.349342 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:10.349349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:10.349354 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:10.349390 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:10.353171 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.355178 (+ 2007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.355185 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.355190 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.355381 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.358184 (+ 2803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.358190 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.358194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.358629 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:10.358909 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.359514 (+ 605us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.359771 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:37:10.360570 12009 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:37:10.370486 11987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b896d54218714c558bd0db8f69cb9c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } } }
I20260810 14:37:10.370754 11987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:10.378661 12009 sys_catalog.cc:455] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b896d54218714c558bd0db8f69cb9c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } } }
I20260810 14:37:10.378985 12009 sys_catalog.cc:458] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:10.382270 12034 catalog_manager.cc:1347] Generated new cluster ID: ef9df81eb8334ab7bfe21d29a7a31388
I20260810 14:37:10.382371 12034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:37:10.395018 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.354300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33776 (local address 127.6.46.252:39801)
0810 14:37:10.354493 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.354498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.362269 (+ 7771us) server_negotiation.cc:407] Connection header received
0810 14:37:10.362477 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.362483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.362584 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.362648 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.366134 (+ 3486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.367223 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.371968 (+ 4745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.372653 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.372744 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.388242 (+ 15498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:10.388249 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:10.388253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:10.388300 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:10.391811 (+ 3511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.392653 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.392659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.392662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.392809 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.393260 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.393265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.393268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.393599 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:10.393830 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.394437 (+ 607us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.394654 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:37:10.399005 11747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.326520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.6.46.254:45003)
0810 14:37:10.326676 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.326681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.349416 (+ 22735us) server_negotiation.cc:407] Connection header received
0810 14:37:10.350380 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.350385 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.350465 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.350535 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.371508 (+ 20973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.372625 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.374522 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.375245 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.375341 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.390175 (+ 14834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:10.390183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:10.390188 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:10.390235 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:10.393853 (+ 3618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.394808 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.394814 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.394817 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.394978 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:10.395524 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:10.395530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:10.395533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:10.397579 (+ 2046us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:10.397825 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.398404 (+ 579us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.398643 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 14:37:10.403435 12034 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:37:10.405287 12034 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:37:10.424057 12034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33: Generated new TSK 0
I20260810 14:37:10.425082 12034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:37:10.428804 12045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b896d54218714c558bd0db8f69cb9c33. Latest consensus state: current_term: 1 leader_uuid: "b896d54218714c558bd0db8f69cb9c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } } }
I20260810 14:37:10.429087 12045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:10.429177 12020 sys_catalog.cc:455] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b896d54218714c558bd0db8f69cb9c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } } }
I20260810 14:37:10.430071 12020 sys_catalog.cc:458] T 00000000000000000000000000000000 P b896d54218714c558bd0db8f69cb9c33 [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:10.429543 12068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b896d54218714c558bd0db8f69cb9c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b896d54218714c558bd0db8f69cb9c33" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 45003 } } peers { permanent_uuid: "4d5bdc5f41144ec49157359734c6ed06" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41321 } } peers { permanent_uuid: "79d4f8e2944b4f3b9c5e2f45b985a428" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 39801 } } }
I20260810 14:37:10.433000 12068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:10.472440 12082 catalog_manager.cc:1259] Loaded cluster ID: ef9df81eb8334ab7bfe21d29a7a31388
I20260810 14:37:10.472625 12082 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06: loading cluster ID for follower catalog manager: success
I20260810 14:37:10.476460 12082 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06: acquiring CA information for follower catalog manager: success
I20260810 14:37:10.478950 12082 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d5bdc5f41144ec49157359734c6ed06: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 14:37:10.576679 11861 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:10.566811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59218 (local address 127.6.46.253:41321)
0810 14:37:10.566985 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.566991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.574614 (+ 7623us) server_negotiation.cc:407] Connection header received
0810 14:37:10.575071 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.575078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.575192 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.575268 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.576435 (+ 1167us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59218: BlockingRecv error: recv got EOF from 127.0.0.1:59218 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 14:37:10.591372 11747 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:10.558680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.6.46.254:45003)
0810 14:37:10.558885 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.558892 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.573973 (+ 15081us) server_negotiation.cc:407] Connection header received
0810 14:37:10.577047 (+ 3074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.577053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.577180 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.577257 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.591114 (+ 13857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47826: BlockingRecv error: recv got EOF from 127.0.0.1:47826 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 14:37:10.594935 11981 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:10.572798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33784 (local address 127.6.46.252:39801)
0810 14:37:10.572965 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.572971 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.592569 (+ 19598us) server_negotiation.cc:407] Connection header received
0810 14:37:10.593571 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.593577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.593657 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.593731 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.594692 (+ 961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33784: BlockingRecv error: recv got EOF from 127.0.0.1:33784 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:37:10.647410 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.631978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59224 (local address 127.6.46.253:41321)
0810 14:37:10.632175 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.632180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.639343 (+ 7163us) server_negotiation.cc:407] Connection header received
0810 14:37:10.639825 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.639831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.640001 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.640129 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:10.640945 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.642194 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.644317 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.644976 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.645118 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.645903 (+ 785us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:10.646605 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:10.646736 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.646978 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.647074 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 14:37:10.694149 11981 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:10.681150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.6.46.252:39801)
0810 14:37:10.681352 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.681358 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.691917 (+ 10559us) server_negotiation.cc:407] Connection header received
0810 14:37:10.692437 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.692443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.692594 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.692671 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.693860 (+ 1189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33792: BlockingRecv error: recv got EOF from 127.0.0.1:33792 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:37:10.747953 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.728775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59236 (local address 127.6.46.253:41321)
0810 14:37:10.728986 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.728992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.741002 (+ 12010us) server_negotiation.cc:407] Connection header received
0810 14:37:10.741300 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.741307 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.741419 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.741514 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:10.742278 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.743400 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.745638 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.746352 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.746448 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.747093 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:10.747264 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:10.747326 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.747556 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.747640 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 14:37:10.770251 11981 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:10.768020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33794 (local address 127.6.46.252:39801)
0810 14:37:10.768200 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.768205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.768665 (+ 460us) server_negotiation.cc:407] Connection header received
0810 14:37:10.768872 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.768908 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.769008 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.769124 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.769970 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33794: BlockingRecv error: recv got EOF from 127.0.0.1:33794 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:37:10.786446 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.779375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.6.46.253:41321)
0810 14:37:10.779548 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.779554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.780080 (+ 526us) server_negotiation.cc:407] Connection header received
0810 14:37:10.780282 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.780291 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.780383 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.780505 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:10.780959 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.782207 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.784102 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.784798 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.784940 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.785422 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:10.785595 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:10.785672 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.785925 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.786064 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260810 14:37:10.797834 11981 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:10.795644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.6.46.252:39801)
0810 14:37:10.795819 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.795825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.796387 (+ 562us) server_negotiation.cc:407] Connection header received
0810 14:37:10.796564 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.796570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.796669 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.796787 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.797606 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33806: BlockingRecv error: recv got EOF from 127.0.0.1:33806 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 14:37:10.814239 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.807540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.6.46.253:41321)
0810 14:37:10.807763 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.807769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.808242 (+ 473us) server_negotiation.cc:407] Connection header received
0810 14:37:10.808419 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.808424 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.808533 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.808645 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:10.809078 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.810321 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.812162 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.812861 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.813005 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.813453 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:10.813600 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:10.813704 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.813829 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.813913 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260810 14:37:10.833685 11981 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:10.831827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33818 (local address 127.6.46.252:39801)
0810 14:37:10.832077 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.832083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.832105 (+ 22us) server_negotiation.cc:407] Connection header received
0810 14:37:10.832398 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.832404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.832513 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.832627 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.833464 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33818: BlockingRecv error: recv got EOF from 127.0.0.1:33818 (error 108)
Metrics: {"server-negotiator.queue_time_us":136}
I20260810 14:37:10.852661 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.845819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.6.46.253:41321)
0810 14:37:10.846008 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.846049 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.846610 (+ 561us) server_negotiation.cc:407] Connection header received
0810 14:37:10.846798 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.846804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.846888 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.846979 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:10.847426 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.848568 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.850513 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.851198 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.851321 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.851842 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:10.851984 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:10.852077 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.852260 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.852341 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 14:37:10.863852 11981 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:10.859866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33824 (local address 127.6.46.252:39801)
0810 14:37:10.860050 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.860056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.862505 (+ 2449us) server_negotiation.cc:407] Connection header received
0810 14:37:10.862641 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.862645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.862751 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.862832 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.863651 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33824: BlockingRecv error: recv got EOF from 127.0.0.1:33824 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 14:37:10.889150 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.883412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59268 (local address 127.6.46.253:41321)
0810 14:37:10.883609 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.883614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.883879 (+ 265us) server_negotiation.cc:407] Connection header received
0810 14:37:10.884063 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.884067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.884134 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.884213 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:10.884677 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.885476 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.887356 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.887804 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.887909 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.888460 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:10.888574 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:10.888647 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.888810 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.888889 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260810 14:37:10.897725 11981 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:10.894764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.6.46.252:39801)
0810 14:37:10.895716 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.895723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.896237 (+ 514us) server_negotiation.cc:407] Connection header received
0810 14:37:10.896508 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.896515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.896616 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.896710 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.897494 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33830: BlockingRecv error: recv got EOF from 127.0.0.1:33830 (error 108)
Metrics: {"server-negotiator.queue_time_us":785}
I20260810 14:37:10.938737 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:10.932935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.6.46.253:41321)
0810 14:37:10.933130 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.933136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.933508 (+ 372us) server_negotiation.cc:407] Connection header received
0810 14:37:10.933715 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.933721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.933812 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.933922 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:10.934307 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.935392 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.936695 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:10.937408 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:10.937547 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:10.937917 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:10.938108 (+ 191us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:10.938203 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:10.938322 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 14:37:10.938412 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260810 14:37:10.948074 11981 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:10.945702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.6.46.252:39801)
0810 14:37:10.945874 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:10.945879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:10.946715 (+ 836us) server_negotiation.cc:407] Connection header received
0810 14:37:10.946862 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:10.946868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:10.946957 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:10.947058 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:10.947851 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33840: BlockingRecv error: recv got EOF from 127.0.0.1:33840 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:37:11.019513 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:11.015126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.6.46.253:41321)
0810 14:37:11.015290 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:11.015295 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:11.015662 (+ 367us) server_negotiation.cc:407] Connection header received
0810 14:37:11.015832 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:11.015836 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:11.015904 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:11.015987 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:11.016278 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.016950 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.018228 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.018612 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.018696 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:11.019000 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:11.019102 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:11.019173 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:11.019243 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 14:37:11.019303 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 14:37:11.025988 11981 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:11.024468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33844 (local address 127.6.46.252:39801)
0810 14:37:11.024611 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:11.024615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:11.024941 (+ 326us) server_negotiation.cc:407] Connection header received
0810 14:37:11.025067 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:11.025070 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:11.025157 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:11.025235 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:11.025830 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33844: BlockingRecv error: recv got EOF from 127.0.0.1:33844 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:37:11.161962 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:11.157405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59296 (local address 127.6.46.253:41321)
0810 14:37:11.157573 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:11.157577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:11.157958 (+ 381us) server_negotiation.cc:407] Connection header received
0810 14:37:11.158140 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:11.158144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:11.158213 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:11.158301 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:11.158670 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.159342 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.160612 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.160999 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.161091 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:11.161393 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:11.161493 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:11.161556 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:11.161653 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 14:37:11.161715 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 14:37:11.164209 12001 catalog_manager.cc:1259] Loaded cluster ID: ef9df81eb8334ab7bfe21d29a7a31388
I20260810 14:37:11.164294 12001 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428: loading cluster ID for follower catalog manager: success
I20260810 14:37:11.167004 12001 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428: acquiring CA information for follower catalog manager: success
I20260810 14:37:11.169410 12001 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 79d4f8e2944b4f3b9c5e2f45b985a428: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:37:11.174400 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:11.167753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33858 (local address 127.6.46.252:39801)
0810 14:37:11.167890 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:11.167894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:11.170058 (+ 2164us) server_negotiation.cc:407] Connection header received
0810 14:37:11.170216 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:11.170220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:11.170364 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:11.170453 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:11.170783 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.171461 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.172689 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.173052 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.173137 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:11.173512 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:11.173934 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:11.174065 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:11.174125 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 14:37:11.174196 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 14:37:11.187448 11861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:11.180319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59302 (local address 127.6.46.253:41321)
0810 14:37:11.180486 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:11.180491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:11.181425 (+ 934us) server_negotiation.cc:407] Connection header received
0810 14:37:11.181924 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:11.181930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:11.182056 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:11.182163 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:11.182831 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.183626 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.185468 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.186151 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.186286 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:11.186713 (+ 427us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:11.186837 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:11.186931 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:11.187072 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 14:37:11.187167 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 14:37:11.188485 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:11.180165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.6.46.254:45003)
0810 14:37:11.180725 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:11.180732 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:11.181402 (+ 670us) server_negotiation.cc:407] Connection header received
0810 14:37:11.181829 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:11.181835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:11.182054 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:11.182200 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:11.183002 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.184071 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.186123 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.186701 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.186805 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:11.187353 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:11.187827 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:11.187935 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:11.188146 (+ 211us) server_negotiation.cc:299] Negotiation successful
0810 14:37:11.188226 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":255,"threads_started":1}
I20260810 14:37:11.188862 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:11.181479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33866 (local address 127.6.46.252:39801)
0810 14:37:11.181624 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:11.181629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:11.182044 (+ 415us) server_negotiation.cc:407] Connection header received
0810 14:37:11.182210 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:11.182215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:11.182304 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:11.182365 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:11.183219 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.184215 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.186257 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:11.186804 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:11.186907 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:11.187564 (+ 657us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:11.187676 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:11.187763 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:11.188560 (+ 797us) server_negotiation.cc:299] Negotiation successful
0810 14:37:11.188631 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 14:37:11.193818 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11631
I20260810 14:37:11.194768 11732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:37:11.301517 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 11631
I20260810 14:37:11.329553 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11754
I20260810 14:37:11.330590 11846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:37:11.434588 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 11754
I20260810 14:37:11.460382 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 11866
I20260810 14:37:11.461462 11965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:37:11.564316 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 11866
2026-08-10T14:37:11Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4403 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 14:37:11 dist-test-slave-q9h9 krb5kdc[12266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 14:37:11 dist-test-slave-q9h9 krb5kdc[12266](info): set up 2 sockets
Aug 10 14:37:11 dist-test-slave-q9h9 krb5kdc[12266](info): commencing operation
krb5kdc: starting...
W20260810 14:37:12.806630 6331 mini_kdc.cc:121] Time spent starting KDC: real 1.198s user 0.003s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/test-user.keytab.
Aug 10 14:37:12 dist-test-slave-q9h9 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T14:37:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T14:37:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.46.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
Entry for principal kudu/127.6.46.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab.
I20260810 14:37:13.080863 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:34457
--webserver_interface=127.6.46.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:34457,127.6.46.253:41467,127.6.46.252:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 14:37:13.405898 12283 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:37:13.406243 12283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:13.406340 12283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:13.406448 12283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:37:13.413414 12283 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:37:13.413491 12283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:13.413581 12283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:13.413643 12283 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:37:13.413725 12283 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:37:13.423101 12283 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:34457,127.6.46.253:41467,127.6.46.252:33455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.254.keytab
--principal=kudu/127.6.46.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.254:34457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.46.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:37:13.424226 12283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:13.425455 12283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:13.436084 12293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:13.436144 12292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:13.436587 12295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:13.437320 12283 server_base.cc:1034] running on GCE node
I20260810 14:37:13.438009 12283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:13.439282 12283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:13.440505 12283 hybrid_clock.cc:648] HybridClock initialized: now 1786372633440436 us; error 40 us; skew 500 ppm
I20260810 14:37:13.440834 12283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:37:13 dist-test-slave-q9h9 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:13.443980 12283 init.cc:377] Logged in from keytab as kudu/127.6.46.254@KRBTEST.COM (short username kudu)
I20260810 14:37:13.445258 12283 webserver.cc:460] Webserver started at http://127.6.46.254:45137/ using document root <none> and password file <none>
I20260810 14:37:13.445923 12283 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:13.446033 12283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:13.446316 12283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:37:13.449055 12283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/data/instance:
uuid: "1b6e954e7e2c41179243a60c4d1e9e54"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-q9h9"
server_key: "bccc44ba74fc0ae9dafbf0bf8b489b86"
server_key_iv: "9f33fe79fb8b386d9cfcef963ae616c0"
server_key_version: "encryptionkey@0"
I20260810 14:37:13.449779 12283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/wal/instance:
uuid: "1b6e954e7e2c41179243a60c4d1e9e54"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-q9h9"
server_key: "bccc44ba74fc0ae9dafbf0bf8b489b86"
server_key_iv: "9f33fe79fb8b386d9cfcef963ae616c0"
server_key_version: "encryptionkey@0"
I20260810 14:37:13.454538 12283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:37:13.457752 12309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:13.458994 12283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 14:37:13.459126 12283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/wal
uuid: "1b6e954e7e2c41179243a60c4d1e9e54"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-q9h9"
server_key: "bccc44ba74fc0ae9dafbf0bf8b489b86"
server_key_iv: "9f33fe79fb8b386d9cfcef963ae616c0"
server_key_version: "encryptionkey@0"
I20260810 14:37:13.459292 12283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:13.476173 12283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:13.480017 12283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:13.480229 12283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:13.496821 12283 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.254:34457
I20260810 14:37:13.496856 12394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.254:34457 every 8 connection(s)
I20260810 14:37:13.498004 12283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/data/info.pb
I20260810 14:37:13.501448 12396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:37:13.506682 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 12283
I20260810 14:37:13.506847 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-0/wal/instance
I20260810 14:37:13.507256 6331 external_mini_cluster.cc:1428] Setting key 96e66e905ed620c3f0d1da95a162b1ac
I20260810 14:37:13.507661 12396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:37:13 dist-test-slave-q9h9 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:37:13.528285 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.510345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:34457 (local address 127.0.0.1:35898)
0810 14:37:13.511088 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.511131 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:13.513033 (+ 1902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:13.513946 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:13.513952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:13.514501 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:13.514853 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:13.514871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:13.516796 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:13.516799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:13.517819 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:13.517831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:13.518916 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:13.518921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:13.519064 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:13.519613 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:13.519637 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:13.521780 (+ 2143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:13.524615 (+ 2835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:13.524626 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:13.524636 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:13.524986 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:13.525351 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:13.525356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:13.525360 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:13.525500 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:13.526206 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:13.526212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:13.526394 (+ 182us) client_negotiation.cc:769] Sending connection context
0810 14:37:13.526600 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 14:37:13.527125 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":126,"thread_start_us":317,"threads_started":1}
I20260810 14:37:13.528319 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.511514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35898 (local address 127.6.46.254:34457)
0810 14:37:13.512156 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:13.512164 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:13.512223 (+ 59us) server_negotiation.cc:407] Connection header received
0810 14:37:13.513393 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:13.513400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:13.513667 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:13.513879 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:13.515069 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:13.516655 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:13.517961 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:13.518723 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:13.518899 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:13.521982 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:13.521990 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:13.522001 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:13.522055 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:13.524497 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:13.525103 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:13.525109 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:13.525111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:13.525251 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:13.525597 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:13.525601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:13.525604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:13.526080 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:13.526249 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:13.526744 (+ 495us) server_negotiation.cc:299] Negotiation successful
0810 14:37:13.526998 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":300,"threads_started":1}
I20260810 14:37:13.531216 12396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:37:13.532176 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.531822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41467 (local address 127.0.0.1:39964)
0810 14:37:13.531973 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.532074 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 14:37:13.532444 12311 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
I20260810 14:37:13.533874 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.533540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41467 (local address 127.0.0.1:39970)
0810 14:37:13.533695 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.533756 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 14:37:13.534065 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:41467: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.6.46.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.253.keytab.
Entry for principal kudu/127.6.46.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.253.keytab.
I20260810 14:37:13.571269 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.253:41467
--webserver_interface=127.6.46.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.253.keytab
--principal=kudu/127.6.46.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:34457,127.6.46.253:41467,127.6.46.252:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:37:13.583381 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } attempt: 1
I20260810 14:37:13.584605 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.584063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41467 (local address 127.0.0.1:39972)
0810 14:37:13.584304 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.584426 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 14:37:13.585606 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.585202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41467 (local address 127.0.0.1:39986)
0810 14:37:13.585376 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.585455 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 14:37:13.585786 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:41467: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
I20260810 14:37:13.654054 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } attempt: 2
I20260810 14:37:13.655616 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.654902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41467 (local address 127.0.0.1:39996)
0810 14:37:13.655310 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.655426 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
I20260810 14:37:13.656795 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.656404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41467 (local address 127.0.0.1:40002)
0810 14:37:13.656553 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.656640 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 14:37:13.657058 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:41467: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
I20260810 14:37:13.748363 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } attempt: 3
I20260810 14:37:13.749579 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.749127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41467 (local address 127.0.0.1:40012)
0810 14:37:13.749339 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.749432 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 14:37:13.750707 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.750345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41467 (local address 127.0.0.1:40022)
0810 14:37:13.750486 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.750565 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 14:37:13.750921 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:41467: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
I20260810 14:37:13.894135 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } attempt: 4
I20260810 14:37:13.895298 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.894786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41467 (local address 127.0.0.1:40028)
0810 14:37:13.895081 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.895169 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260810 14:37:13.896215 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.895873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41467 (local address 127.0.0.1:40044)
0810 14:37:13.896016 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.896098 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 14:37:13.896389 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.253:41467: Network error: Client connection negotiation failed: client connection to 127.6.46.253:41467: connect: Connection refused (error 111)
W20260810 14:37:13.897459 12413 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:37:13.897749 12413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:13.897840 12413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:13.897946 12413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:37:13.905118 12413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:37:13.905212 12413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:13.905328 12413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:13.905395 12413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:37:13.905478 12413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:37:13.915637 12413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:34457,127.6.46.253:41467,127.6.46.252:33455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.253.keytab
--principal=kudu/127.6.46.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.253:41467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.46.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:37:13.916797 12413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:13.918061 12413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:13.929097 12423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:13.929143 12424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:13.929169 12426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:13.929924 12413 server_base.cc:1034] running on GCE node
I20260810 14:37:13.930641 12413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:13.931876 12413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:13.933090 12413 hybrid_clock.cc:648] HybridClock initialized: now 1786372633933027 us; error 37 us; skew 500 ppm
I20260810 14:37:13.933387 12413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:37:13 dist-test-slave-q9h9 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:13.936445 12413 init.cc:377] Logged in from keytab as kudu/127.6.46.253@KRBTEST.COM (short username kudu)
I20260810 14:37:13.937731 12413 webserver.cc:460] Webserver started at http://127.6.46.253:34699/ using document root <none> and password file <none>
I20260810 14:37:13.938378 12413 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:13.938465 12413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:13.938723 12413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:37:13.941424 12413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/data/instance:
uuid: "c99f72cf93764d6b8ea2d4d7a49ff000"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-q9h9"
server_key: "6fa27b767562bf660ed84c4af16efa7e"
server_key_iv: "b7eb98639fb1fe26f9465e384db73451"
server_key_version: "encryptionkey@0"
I20260810 14:37:13.942163 12413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/wal/instance:
uuid: "c99f72cf93764d6b8ea2d4d7a49ff000"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-q9h9"
server_key: "6fa27b767562bf660ed84c4af16efa7e"
server_key_iv: "b7eb98639fb1fe26f9465e384db73451"
server_key_version: "encryptionkey@0"
I20260810 14:37:13.946813 12413 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 14:37:13.949903 12435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:13.951109 12413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 14:37:13.951254 12413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/wal
uuid: "c99f72cf93764d6b8ea2d4d7a49ff000"
format_stamp: "Formatted at 2026-08-10 14:37:13 on dist-test-slave-q9h9"
server_key: "6fa27b767562bf660ed84c4af16efa7e"
server_key_iv: "b7eb98639fb1fe26f9465e384db73451"
server_key_version: "encryptionkey@0"
I20260810 14:37:13.951418 12413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:13.963501 12413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:13.967401 12413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:13.967612 12413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:13.984570 12413 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.253:41467
I20260810 14:37:13.984587 12521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.253:41467 every 8 connection(s)
I20260810 14:37:13.985863 12413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/data/info.pb
I20260810 14:37:13.988338 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 12413
I20260810 14:37:13.988511 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-1/wal/instance
I20260810 14:37:13.988909 6331 external_mini_cluster.cc:1428] Setting key 4588515c5f48954c24f26660db44d054
I20260810 14:37:13.989581 12522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:37:13.995908 12522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:37:14 dist-test-slave-q9h9 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.253@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:37:14.016060 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.998327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35910 (local address 127.6.46.254:34457)
0810 14:37:13.998537 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:13.998543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:13.999740 (+ 1197us) server_negotiation.cc:407] Connection header received
0810 14:37:14.001986 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.001991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.002085 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.002198 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.003842 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.004812 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.006298 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.006842 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.006959 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.010414 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.010419 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.010423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.010457 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.013008 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.013781 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.013784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.013786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.013903 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.014420 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.014423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.014425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.014726 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.014840 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.015668 (+ 828us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.015837 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 14:37:14.017364 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:13.998790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:34457 (local address 127.0.0.1:35910)
0810 14:37:13.999551 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:13.999606 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.001724 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.002417 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.002427 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.003123 (+ 696us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:14.003668 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.003686 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.004939 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.004943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.006155 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.006167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.007018 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.007022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.007234 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.007835 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:14.007861 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:14.010201 (+ 2340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:14.013153 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.013162 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.013176 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.013688 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.014005 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.014010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.014035 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.014322 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.014848 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:14.014855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:14.015263 (+ 408us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.015586 (+ 323us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.016065 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":308,"threads_started":1}
I20260810 14:37:14.019080 12522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.46.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.46.252@KRBTEST.COM" created.
Aug 10 14:37:14 dist-test-slave-q9h9 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.253@KRBTEST.COM for kudu/127.6.46.253@KRBTEST.COM
I20260810 14:37:14.033279 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.019878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41467 (local address 127.0.0.1:40050)
0810 14:37:14.020076 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.020094 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.020210 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.020930 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.020934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.021375 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:14.021512 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.021520 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.022878 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.022882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.023758 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.023770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.024691 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.024697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.024862 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.025408 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:14.025428 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:14.027591 (+ 2163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:14.031021 (+ 3430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.031025 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.031027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.031374 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.031805 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.031810 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.031814 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.031966 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.032601 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:14.032604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:14.032734 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.032840 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.033019 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 14:37:14.033619 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.019972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.6.46.253:41467)
0810 14:37:14.020532 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.020537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.020563 (+ 26us) server_negotiation.cc:407] Connection header received
0810 14:37:14.020603 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.020608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.020771 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.020876 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.021658 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.022752 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.023905 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.024515 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.024619 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.027797 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.027809 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.027824 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.027862 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.030903 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.031496 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.031504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.031507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.031672 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.032073 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.032077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.032080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.032480 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.032687 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.032990 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.033216 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":283,"threads_started":1}
I20260810 14:37:14.035928 12522 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:37:14.037011 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.036606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54310)
0810 14:37:14.036775 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.036879 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 14:37:14.037273 12436 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
I20260810 14:37:14.038776 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.038428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54312)
0810 14:37:14.038562 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.038649 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 14:37:14.039042 12522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:33455: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.46.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.252.keytab.
Entry for principal kudu/127.6.46.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.252.keytab.
I20260810 14:37:14.059381 6331 external_mini_cluster.cc:1326] Running /tmp/dist-test-task50NkZQ/build/asan/bin/kudu
/tmp/dist-test-task50NkZQ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.252:33455
--webserver_interface=127.6.46.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.252.keytab
--principal=kudu/127.6.46.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.46.212:35135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.46.254:34457,127.6.46.253:41467,127.6.46.252:33455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 14:37:14.088330 12522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } attempt: 1
I20260810 14:37:14.089476 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.088935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54322)
0810 14:37:14.089164 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.089292 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 14:37:14.090556 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.090234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54330)
0810 14:37:14.090366 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.090456 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 14:37:14.090787 12522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:33455: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
I20260810 14:37:14.159015 12522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } attempt: 2
I20260810 14:37:14.160153 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.159702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54346)
0810 14:37:14.159895 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.159994 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 14:37:14.161129 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.160786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54352)
0810 14:37:14.160948 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.161015 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260810 14:37:14.161343 12522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:33455: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
I20260810 14:37:14.195649 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } attempt: 5
Aug 10 14:37:14 dist-test-slave-q9h9 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for kudu/127.6.46.253@KRBTEST.COM
I20260810 14:37:14.209824 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.196372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41467 (local address 127.0.0.1:40062)
0810 14:37:14.196562 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.196583 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.196703 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.197051 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.197054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.197451 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:14.197625 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.197634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.199108 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.199113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.199999 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.200012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.200958 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.200962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.201097 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.201745 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:14.201763 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:14.204054 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:14.207663 (+ 3609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.207667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.207670 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.208050 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.208416 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.208419 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.208421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.208546 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.209106 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:14.209109 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:14.209254 (+ 145us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.209361 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.209579 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 14:37:14.210103 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.196473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40062 (local address 127.6.46.253:41467)
0810 14:37:14.196695 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.196701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.196729 (+ 28us) server_negotiation.cc:407] Connection header received
0810 14:37:14.196795 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.196800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.196898 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.197016 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.197792 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.198945 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.200154 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.200824 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.200956 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.204250 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.204256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.204261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.204300 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.207519 (+ 3219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.208163 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.208168 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.208172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.208319 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.208648 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.208653 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.208656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.209005 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.209159 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.209494 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.209756 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 14:37:14.211145 12396 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } has no permanent_uuid. Determining permanent_uuid...
I20260810 14:37:14.212237 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.211867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54368)
0810 14:37:14.212021 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.212090 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 14:37:14.213128 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.212843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54376)
0810 14:37:14.212962 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.213038 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 14:37:14.213336 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:33455: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
I20260810 14:37:14.252548 12522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } attempt: 3
I20260810 14:37:14.253753 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.253278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54386)
0810 14:37:14.253486 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.253583 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 14:37:14.254907 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.254467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54392)
0810 14:37:14.254690 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.254792 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
W20260810 14:37:14.255152 12522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:33455: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
I20260810 14:37:14.264540 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } attempt: 1
I20260810 14:37:14.265453 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.265059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54402)
0810 14:37:14.265223 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.265304 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260810 14:37:14.266360 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.266073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54406)
0810 14:37:14.266203 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.266266 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 14:37:14.266549 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:33455: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
I20260810 14:37:14.334748 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } attempt: 2
I20260810 14:37:14.335836 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.335344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54410)
0810 14:37:14.335574 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.335662 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260810 14:37:14.336822 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.336487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54414)
0810 14:37:14.336657 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.336720 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 14:37:14.337072 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:33455: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
W20260810 14:37:14.383494 12532 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 14:37:14.383800 12532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 14:37:14.383898 12532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 14:37:14.384009 12532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 14:37:14.390859 12532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 14:37:14.390944 12532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 14:37:14.391029 12532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 14:37:14.391093 12532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 14:37:14.391187 12532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 14:37:14.398380 12522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } attempt: 4
I20260810 14:37:14.399519 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.399036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54416)
0810 14:37:14.399253 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.399350 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 14:37:14.400552 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.400186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54430)
0810 14:37:14.400359 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.400432 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 14:37:14.400779 12522 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:33455: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
I20260810 14:37:14.400985 12532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.46.212:35135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.46.254:34457,127.6.46.253:41467,127.6.46.252:33455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/krb5kdc/kudu_127.6.46.252.keytab
--principal=kudu/127.6.46.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.46.252:33455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.46.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 13:46:55 UTC on ccycloud.quasar-skomnt.root.comops.site
build id 4577964
ASAN enabled
I20260810 14:37:14.402220 12532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 14:37:14.403431 12532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 14:37:14.414333 12543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:14.414340 12541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 14:37:14.414458 12540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 14:37:14.415390 12532 server_base.cc:1034] running on GCE node
I20260810 14:37:14.416076 12532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 14:37:14.417387 12532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 14:37:14.418596 12532 hybrid_clock.cc:648] HybridClock initialized: now 1786372634418537 us; error 31 us; skew 500 ppm
I20260810 14:37:14.418889 12532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 14:37:14 dist-test-slave-q9h9 krb5kdc[12266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372634, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 14:37:14.421824 12532 init.cc:377] Logged in from keytab as kudu/127.6.46.252@KRBTEST.COM (short username kudu)
I20260810 14:37:14.423178 12532 webserver.cc:460] Webserver started at http://127.6.46.252:45009/ using document root <none> and password file <none>
I20260810 14:37:14.423815 12532 fs_manager.cc:359] Metadata directory not provided
I20260810 14:37:14.423907 12532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 14:37:14.424224 12532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 14:37:14.427150 12532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/data/instance:
uuid: "40f64b7412904560a5770681e7fb486e"
format_stamp: "Formatted at 2026-08-10 14:37:14 on dist-test-slave-q9h9"
server_key: "09e171830ec4ea058af315720354dc5e"
server_key_iv: "792557aa0a9092bdcd21aed142701227"
server_key_version: "encryptionkey@0"
I20260810 14:37:14.427884 12532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/wal/instance:
uuid: "40f64b7412904560a5770681e7fb486e"
format_stamp: "Formatted at 2026-08-10 14:37:14 on dist-test-slave-q9h9"
server_key: "09e171830ec4ea058af315720354dc5e"
server_key_iv: "792557aa0a9092bdcd21aed142701227"
server_key_version: "encryptionkey@0"
I20260810 14:37:14.432780 12532 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 14:37:14.436048 12557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 14:37:14.437317 12532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 14:37:14.437474 12532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/data,/tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/wal
uuid: "40f64b7412904560a5770681e7fb486e"
format_stamp: "Formatted at 2026-08-10 14:37:14 on dist-test-slave-q9h9"
server_key: "09e171830ec4ea058af315720354dc5e"
server_key_iv: "792557aa0a9092bdcd21aed142701227"
server_key_version: "encryptionkey@0"
I20260810 14:37:14.437673 12532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 14:37:14.443284 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } attempt: 3
I20260810 14:37:14.444298 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.443869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54442)
0810 14:37:14.444060 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.444144 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 14:37:14.445261 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.444896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54444)
0810 14:37:14.445094 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.445163 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260810 14:37:14.445465 12396 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.46.252:33455: Network error: Client connection negotiation failed: client connection to 127.6.46.252:33455: connect: Connection refused (error 111)
I20260810 14:37:14.450632 12532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 14:37:14.454543 12532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 14:37:14.454807 12532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 14:37:14.471378 12532 rpc_server.cc:307] RPC server started. Bound to: 127.6.46.252:33455
I20260810 14:37:14.471397 12637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.46.252:33455 every 8 connection(s)
I20260810 14:37:14.472595 12532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/data/info.pb
I20260810 14:37:14.475486 6331 external_mini_cluster.cc:1388] Started /tmp/dist-test-task50NkZQ/build/asan/bin/kudu as pid 12532
I20260810 14:37:14.475668 6331 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task50NkZQ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786372275129649-6331-0/minicluster-data/master-2/wal/instance
I20260810 14:37:14.476080 6331 external_mini_cluster.cc:1428] Setting key 23cb5ba924eec02fa0d93f58297ef674
I20260810 14:37:14.476114 12638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 14:37:14.485785 12638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:37:14 dist-test-slave-q9h9 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.252@KRBTEST.COM
Aug 10 14:37:14 dist-test-slave-q9h9 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372634, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.252@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
I20260810 14:37:14.499123 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.477638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.6.46.252:33455)
0810 14:37:14.478442 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.478457 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.478508 (+ 51us) server_negotiation.cc:407] Connection header received
0810 14:37:14.480789 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.480817 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.481298 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.481712 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.483102 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.485012 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.486185 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.487165 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.487330 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.491480 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.491490 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.491503 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.491551 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.494635 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.495383 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.495389 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.495394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.495556 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.495917 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.495923 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.495926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.496597 (+ 671us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.496894 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.497323 (+ 429us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.497862 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":356,"threads_started":1}
I20260810 14:37:14.503625 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.488339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.254:34457 (local address 127.0.0.1:35912)
0810 14:37:14.489046 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.489085 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.489240 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.489607 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.489613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.490340 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:14.490652 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.490665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.491724 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.491728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.492902 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.492914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.493856 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.493861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.493983 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.494508 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:14.494524 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:14.497048 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:14.500885 (+ 3837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.500896 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.500903 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.501325 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.501760 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.501765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.501768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.501933 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.502686 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:14.502691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:14.502817 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.502957 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.503246 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":361,"threads_started":1}
I20260810 14:37:14.503719 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.488356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.6.46.254:34457)
0810 14:37:14.488558 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.488564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.489195 (+ 631us) server_negotiation.cc:407] Connection header received
0810 14:37:14.489377 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.489382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.489462 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.489578 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.490797 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.491558 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.493037 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.493681 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.493811 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.497273 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.497280 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.497285 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.497321 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.500767 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.501455 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.501460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.501463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.501636 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.502116 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.502120 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.502123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.502448 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.502594 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.503103 (+ 509us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.503327 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:37:14.505831 12638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:37:14 dist-test-slave-q9h9 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.254@KRBTEST.COM
Aug 10 14:37:14 dist-test-slave-q9h9 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372634, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.252@KRBTEST.COM for kudu/127.6.46.253@KRBTEST.COM
I20260810 14:37:14.517491 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.502107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.6.46.254:34457)
0810 14:37:14.503982 (+ 1875us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.503987 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.504001 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:37:14.504046 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.504050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.504109 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.504197 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.505641 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.506726 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.507899 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.508296 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.508402 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.512292 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.512296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.512299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.512333 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.515006 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.515722 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.515725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.515728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.515828 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.516183 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.516187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.516189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.516438 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.516695 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.516949 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.517139 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1786,"thread_start_us":3667,"threads_started":1}
I20260810 14:37:14.517915 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.506567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.253:41467 (local address 127.0.0.1:40066)
0810 14:37:14.506851 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.506864 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.506957 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.507234 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.507238 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.507579 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:14.507713 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.507721 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.508617 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.508623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.509563 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.509575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.510307 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.510313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.510479 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.510912 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:14.510926 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:14.512893 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:14.515815 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.515820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.515823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.516149 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.516482 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.516487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.516491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.516624 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.517077 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:14.517081 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:14.517222 (+ 141us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.517394 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.517617 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 14:37:14.517982 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.506608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40066 (local address 127.6.46.253:41467)
0810 14:37:14.506776 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.506780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.506929 (+ 149us) server_negotiation.cc:407] Connection header received
0810 14:37:14.507051 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.507056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.507126 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.507204 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.507847 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.508484 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.509707 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.510166 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.510266 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.513056 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.513060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.513063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.513093 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.515707 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.516251 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.516256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.516260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.516384 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.516718 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.516723 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.516727 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.516960 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.517094 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.517517 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.517727 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 14:37:14.518857 12638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 14:37:14 dist-test-slave-q9h9 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372634, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.252@KRBTEST.COM for kudu/127.6.46.252@KRBTEST.COM
Aug 10 14:37:14 dist-test-slave-q9h9 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.46.253@KRBTEST.COM
I20260810 14:37:14.531131 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.519561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54462)
0810 14:37:14.519761 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.519773 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.519846 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.520100 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.520103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.520404 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:14.520518 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.520526 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.521494 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.521500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.522422 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.522434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.523251 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.523257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.523465 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.523940 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:14.523958 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:14.525641 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:14.528787 (+ 3146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.528792 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.528796 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.529199 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.529561 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.529564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.529566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.529687 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.530265 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:14.530268 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:14.530379 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.530502 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.530773 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260810 14:37:14.531195 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.519523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.6.46.252:33455)
0810 14:37:14.519674 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.519678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.519831 (+ 153us) server_negotiation.cc:407] Connection header received
0810 14:37:14.519917 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.519921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.519990 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.520099 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.520652 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.521378 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.522552 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.523104 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.523193 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.525811 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.525818 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.525823 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.525860 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.528664 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.529310 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.529315 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.529319 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.529450 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.529789 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.529795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.529799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.530161 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.530304 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.530585 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.530894 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":125,"server-negotiator.queue_time_us":55}
I20260810 14:37:14.532821 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.519080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.6.46.253:41467)
0810 14:37:14.519233 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.519238 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.519256 (+ 18us) server_negotiation.cc:407] Connection header received
0810 14:37:14.519376 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.519380 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.519444 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.519529 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.520916 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.521630 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.522880 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.523304 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.523412 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.527372 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.527378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.527382 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.527417 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.530375 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.531062 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.531067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.531071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.531211 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.531668 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.531673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.531677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.531913 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.532147 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.532435 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.532584 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:37:14.535923 12638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e: Bootstrap starting.
I20260810 14:37:14.536969 6331 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 14:37:14.539242 12638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:14.540635 12638 log.cc:824] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:14.539307 12347 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1b6e954e7e2c41179243a60c4d1e9e54"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35920
I20260810 14:37:14.543606 12638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e: No bootstrap required, opened a new log
I20260810 14:37:14.554230 12638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } }
I20260810 14:37:14.554623 12638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40f64b7412904560a5770681e7fb486e, State: Initialized, Role: FOLLOWER
I20260810 14:37:14.555157 12638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } }
I20260810 14:37:14.556063 12648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } } }
I20260810 14:37:14.556469 12648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:14.557117 12638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e [sys.catalog]: configured and running, proceeding with master startup.
W20260810 14:37:14.565876 12669 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 14:37:14.566056 12669 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 14:37:14.563930 12532 master_runner.cc:186] Error getting master registration for 127.6.46.254:34457: OK, instance_id { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" instance_seqno: 1786372633492064 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:37:14.571521 12532 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:37:14.622668 12396 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } attempt: 4
Aug 10 14:37:14 dist-test-slave-q9h9 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.254@KRBTEST.COM for kudu/127.6.46.252@KRBTEST.COM
I20260810 14:37:14.638767 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.623326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54476)
0810 14:37:14.623553 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.623575 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.623710 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.624163 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.624168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.624724 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:14.624946 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.624960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.627058 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.627063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.628505 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.628523 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.629252 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.629257 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.629453 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.630042 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:14.630068 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:14.632749 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:14.636095 (+ 3346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.636100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.636105 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.636593 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.636961 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.636966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.636970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.637131 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.637740 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:14.637744 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:14.637903 (+ 159us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.638083 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.638368 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 14:37:14.639605 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.623430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.6.46.252:33455)
0810 14:37:14.623606 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.623617 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.623636 (+ 19us) server_negotiation.cc:407] Connection header received
0810 14:37:14.623913 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.623919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.623998 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.624092 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.625093 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.626909 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.628653 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.629101 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.629220 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.632931 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.632936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.632940 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.632970 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.635946 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.636713 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.636719 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.636723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.636852 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.637234 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.637240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.637243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.637615 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.637775 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.639051 (+ 1276us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.639272 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 14:37:14.645682 12396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54: Bootstrap starting.
I20260810 14:37:14.650283 12396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:14.651862 12396 log.cc:824] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:14.655541 12396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54: No bootstrap required, opened a new log
I20260810 14:37:14.663396 12396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } }
I20260810 14:37:14.663901 12396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b6e954e7e2c41179243a60c4d1e9e54, State: Initialized, Role: FOLLOWER
I20260810 14:37:14.664528 12396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } }
I20260810 14:37:14.666535 12673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } } }
I20260810 14:37:14.666968 12673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:14.667883 12396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:37:14.668004 12347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 14:37:14.668215 12347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:14.671726 12347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } }
I20260810 14:37:14.673491 12347 leader_election.cc:288] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [CANDIDATE]: Term 1 election: Requested vote from peers c99f72cf93764d6b8ea2d4d7a49ff000 (127.6.46.253:41467), 40f64b7412904560a5770681e7fb486e (127.6.46.252:33455)
I20260810 14:37:14.674240 12604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "40f64b7412904560a5770681e7fb486e"
I20260810 14:37:14.674765 12604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:14.678347 12604 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b6e954e7e2c41179243a60c4d1e9e54 in term 1.
I20260810 14:37:14.679088 12310 leader_election.cc:302] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b6e954e7e2c41179243a60c4d1e9e54, 40f64b7412904560a5770681e7fb486e; no voters:
I20260810 14:37:14.679509 12673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 14:37:14.680567 12673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [term 1 LEADER]: Becoming Leader. State: Replica: 1b6e954e7e2c41179243a60c4d1e9e54, State: Running, Role: LEADER
I20260810 14:37:14.681206 12673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } }
I20260810 14:37:14.689810 12688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b6e954e7e2c41179243a60c4d1e9e54. Latest consensus state: current_term: 1 leader_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } } }
I20260810 14:37:14.690546 12688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:14.699915 12522 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } attempt: 5
W20260810 14:37:14.706688 12575 tablet.cc:2369] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 14:37:14 dist-test-slave-q9h9 krb5kdc[12266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786372633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.46.253@KRBTEST.COM for kudu/127.6.46.252@KRBTEST.COM
W20260810 14:37:14.732770 12315 tablet.cc:2369] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 14:37:14.734758 12700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 14:37:14.736809 12524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.700442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.46.252:33455 (local address 127.0.0.1:54484)
0810 14:37:14.700683 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 14:37:14.700701 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 14:37:14.700812 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 14:37:14.710902 (+ 10090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 14:37:14.710908 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 14:37:14.711447 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0810 14:37:14.711676 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.711703 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.713541 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.713547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.715020 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 14:37:14.715037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.716070 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.716076 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 14:37:14.716302 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.717022 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 14:37:14.717046 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 14:37:14.720122 (+ 3076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 14:37:14.734125 (+ 14003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.734131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.734136 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.734654 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.735127 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 14:37:14.735132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 14:37:14.735136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 14:37:14.735321 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 14:37:14.735860 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 14:37:14.735865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 14:37:14.736017 (+ 152us) client_negotiation.cc:769] Sending connection context
0810 14:37:14.736167 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 14:37:14.736438 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20260810 14:37:14.736842 12708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.701098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54484 (local address 127.6.46.252:33455)
0810 14:37:14.710503 (+ 9405us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.710509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.710537 (+ 28us) server_negotiation.cc:407] Connection header received
0810 14:37:14.710603 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.710609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.710699 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.711858 (+ 1159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.712114 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.713360 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.715222 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.715891 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.720294 (+ 4403us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.720534 (+ 240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.720541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.720546 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.720577 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.726958 (+ 6381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.734828 (+ 7870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.734833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.734837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.734986 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.735412 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.735417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.735420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.735745 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.735867 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.736255 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.736437 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9275,"thread_start_us":9104,"threads_started":1}
I20260810 14:37:14.743327 12522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000: Bootstrap starting.
I20260810 14:37:14.749039 12522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000: Neither blocks nor log segments found. Creating new log.
I20260810 14:37:14.750902 12522 log.cc:824] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000: Log is configured to *not* fsync() on all Append() calls
I20260810 14:37:14.764994 12522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000: No bootstrap required, opened a new log
I20260810 14:37:14.765354 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.713897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35932 (local address 127.6.46.254:34457)
0810 14:37:14.714086 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.714092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.727178 (+ 13086us) server_negotiation.cc:407] Connection header received
0810 14:37:14.727429 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.727435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.727519 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.727599 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.729727 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.730880 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.733121 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.733891 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.733995 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.752877 (+ 18882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.752885 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.752891 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.752965 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.756783 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.759151 (+ 2368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.759157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.759160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.759317 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.763051 (+ 3734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.763056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.763059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.763386 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.763679 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.764304 (+ 625us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.764692 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:37:14.765578 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.700343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.6.46.252:33455)
0810 14:37:14.700520 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.700526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.721634 (+ 21108us) server_negotiation.cc:407] Connection header received
0810 14:37:14.721862 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.721868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.721964 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.722054 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.724247 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.725321 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.738871 (+ 13550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.739609 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.739704 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.748845 (+ 9141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.748853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.748859 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.748898 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.752628 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.760493 (+ 7865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.760500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.760502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.760658 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.762855 (+ 2197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.762862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.762865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.763517 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.763948 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.765029 (+ 1081us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.765242 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:37:14.771366 12700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 14:37:14.776453 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:14.696288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40088 (local address 127.6.46.253:41467)
0810 14:37:14.696484 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:14.696491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:14.715289 (+ 18798us) server_negotiation.cc:407] Connection header received
0810 14:37:14.715748 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:14.715754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:14.715832 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:14.715898 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:14.718093 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.719273 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.737130 (+ 17857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:14.737799 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:14.737904 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:14.758384 (+ 20480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 14:37:14.758391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 14:37:14.758395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 14:37:14.758432 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 14:37:14.762280 (+ 3848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.774125 (+ 11845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.774130 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.774133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.774290 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 14:37:14.774762 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 14:37:14.774767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 14:37:14.774770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 14:37:14.775122 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 14:37:14.775362 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:14.775886 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 14:37:14.776108 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 14:37:14.778630 12522 raft_consensus.cc:348] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } }
I20260810 14:37:14.778988 12522 raft_consensus.cc:729] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99f72cf93764d6b8ea2d4d7a49ff000, State: Initialized, Role: FOLLOWER
I20260810 14:37:14.779484 12522 consensus_queue.cc:260] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } }
I20260810 14:37:14.790527 12718 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } } }
I20260810 14:37:14.790951 12718 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:14.791783 12522 sys_catalog.cc:564] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 14:37:14.791852 12486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000"
I20260810 14:37:14.792333 12486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000 [term 0 FOLLOWER]: Advancing to term 1
I20260810 14:37:14.795816 12486 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b6e954e7e2c41179243a60c4d1e9e54 in term 1.
I20260810 14:37:14.805609 12604 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e [term 1 FOLLOWER]: Refusing update from remote peer 1b6e954e7e2c41179243a60c4d1e9e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:37:14.806458 12486 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000 [term 1 FOLLOWER]: Refusing update from remote peer 1b6e954e7e2c41179243a60c4d1e9e54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 14:37:14.806541 12673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:37:14.807204 12673 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 14:37:14.802738 12283 master_runner.cc:186] Error getting master registration for 127.6.46.253:41467: OK, instance_id { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" instance_seqno: 1786372633979599 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 14:37:14.821373 12283 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 14:37:14.823386 12648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b6e954e7e2c41179243a60c4d1e9e54. Latest consensus state: current_term: 1 leader_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } } }
I20260810 14:37:14.823653 12648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:14.825361 12688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } } }
I20260810 14:37:14.825603 12688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:14.828809 12731 mvcc.cc:205] Tried to move back new op lower bound from 7316982312151834624 to 7316982311659237376. Current Snapshot: MvccSnapshot[applied={T|T < 7316982312151834624}]
I20260810 14:37:14.839144 12718 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b6e954e7e2c41179243a60c4d1e9e54. Latest consensus state: current_term: 1 leader_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } } }
I20260810 14:37:14.839402 12718 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:14.839463 12648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } } }
I20260810 14:37:14.839680 12648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:14.841419 12700 catalog_manager.cc:1347] Generated new cluster ID: a95d15fc07d2409fa6d7ec7c2104a3f5
I20260810 14:37:14.841534 12700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 14:37:14.846930 12673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } } }
I20260810 14:37:14.847218 12673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54 [sys.catalog]: This master's current role is: LEADER
I20260810 14:37:14.850307 12734 mvcc.cc:205] Tried to move back new op lower bound from 7316982312151834624 to 7316982311659237376. Current Snapshot: MvccSnapshot[applied={T|T < 7316982312151834624}]
I20260810 14:37:14.855082 12718 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b6e954e7e2c41179243a60c4d1e9e54" member_type: VOTER last_known_addr { host: "127.6.46.254" port: 34457 } } peers { permanent_uuid: "c99f72cf93764d6b8ea2d4d7a49ff000" member_type: VOTER last_known_addr { host: "127.6.46.253" port: 41467 } } peers { permanent_uuid: "40f64b7412904560a5770681e7fb486e" member_type: VOTER last_known_addr { host: "127.6.46.252" port: 33455 } } }
I20260810 14:37:14.855306 12718 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 14:37:14.868526 12700 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 14:37:14.870141 12700 catalog_manager.cc:1504] Loading token signing keys...
I20260810 14:37:14.893513 12700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b6e954e7e2c41179243a60c4d1e9e54: Generated new TSK 0
I20260810 14:37:14.894487 12700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 14:37:14.905778 12748 catalog_manager.cc:1259] Loaded cluster ID: a95d15fc07d2409fa6d7ec7c2104a3f5
I20260810 14:37:14.905872 12748 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000: loading cluster ID for follower catalog manager: success
I20260810 14:37:14.920872 12748 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000: acquiring CA information for follower catalog manager: success
I20260810 14:37:14.923122 12748 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c99f72cf93764d6b8ea2d4d7a49ff000: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 14:37:15.072461 12526 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:15.046788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40098 (local address 127.6.46.253:41467)
0810 14:37:15.046982 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.046993 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.069400 (+ 22407us) server_negotiation.cc:407] Connection header received
0810 14:37:15.070968 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.070974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.071087 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.071157 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:15.072228 (+ 1071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40098: BlockingRecv error: recv got EOF from 127.0.0.1:40098 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260810 14:37:15.074290 12643 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:15.053479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.6.46.252:33455)
0810 14:37:15.053658 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.053663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.070558 (+ 16895us) server_negotiation.cc:407] Connection header received
0810 14:37:15.072913 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.072918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.072991 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.073050 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:15.073988 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54486: BlockingRecv error: recv got EOF from 127.0.0.1:54486 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 14:37:15.076750 12406 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:15.067617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.6.46.254:34457)
0810 14:37:15.067840 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.067847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.074921 (+ 7074us) server_negotiation.cc:407] Connection header received
0810 14:37:15.075093 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.075098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.075183 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.075279 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:15.076513 (+ 1234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35940: BlockingRecv error: recv got EOF from 127.0.0.1:35940 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 14:37:15.124567 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.106784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.6.46.253:41467)
0810 14:37:15.106954 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.106960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.117515 (+ 10555us) server_negotiation.cc:407] Connection header received
0810 14:37:15.117987 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.117994 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.118244 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.118377 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.119017 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.120091 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.121821 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.122407 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.122492 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.123133 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.123804 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.123917 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.124143 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.124247 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 14:37:15.133441 12643 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:15.131562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54496 (local address 127.6.46.252:33455)
0810 14:37:15.131746 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.131752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.132166 (+ 414us) server_negotiation.cc:407] Connection header received
0810 14:37:15.132348 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.132353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.132440 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.132527 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:15.133235 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54496: BlockingRecv error: recv got EOF from 127.0.0.1:54496 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 14:37:15.142373 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.137137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40114 (local address 127.6.46.253:41467)
0810 14:37:15.137323 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.137329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.137770 (+ 441us) server_negotiation.cc:407] Connection header received
0810 14:37:15.137931 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.137934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.138010 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.138116 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.138589 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.139333 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.140585 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.141079 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.141201 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.141545 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.141697 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.141785 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.141977 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.142079 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 14:37:15.149603 12643 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:15.148201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.6.46.252:33455)
0810 14:37:15.148379 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.148385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.148400 (+ 15us) server_negotiation.cc:407] Connection header received
0810 14:37:15.148459 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.148463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.148538 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.148628 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:15.149368 (+ 740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54506: BlockingRecv error: recv got EOF from 127.0.0.1:54506 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:37:15.159651 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.154235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40130 (local address 127.6.46.253:41467)
0810 14:37:15.154416 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.154422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.154858 (+ 436us) server_negotiation.cc:407] Connection header received
0810 14:37:15.155035 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.155041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.155128 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.155216 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.155675 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.156460 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.157870 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.158416 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.158529 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.158918 (+ 389us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.159034 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.159110 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.159263 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.159335 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 14:37:15.166708 12643 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:15.165206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54522 (local address 127.6.46.252:33455)
0810 14:37:15.165360 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.165366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.165383 (+ 17us) server_negotiation.cc:407] Connection header received
0810 14:37:15.165598 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.165604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.165687 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.165769 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:15.166504 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54522: BlockingRecv error: recv got EOF from 127.0.0.1:54522 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 14:37:15.178296 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.173206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40138 (local address 127.6.46.253:41467)
0810 14:37:15.173440 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.173446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.173888 (+ 442us) server_negotiation.cc:407] Connection header received
0810 14:37:15.174077 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.174081 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.174149 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.174223 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.174594 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.175333 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.176614 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.177113 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.177215 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.177621 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.177727 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.177805 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.177940 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.178041 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
W20260810 14:37:15.193460 12643 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:15.183550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.6.46.252:33455)
0810 14:37:15.183830 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.183837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.191996 (+ 8159us) server_negotiation.cc:407] Connection header received
0810 14:37:15.192170 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.192175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.192293 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.192378 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:15.193228 (+ 850us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54538: BlockingRecv error: recv got EOF from 127.0.0.1:54538 (error 108)
Metrics: {"server-negotiator.queue_time_us":164}
I20260810 14:37:15.209079 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.203893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40148 (local address 127.6.46.253:41467)
0810 14:37:15.204098 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.204104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.204547 (+ 443us) server_negotiation.cc:407] Connection header received
0810 14:37:15.204707 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.204712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.204780 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.204854 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.205216 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.205940 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.207277 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.207727 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.207833 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.208304 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.208444 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.208496 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.208682 (+ 186us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.208759 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 14:37:15.216277 12643 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:15.214268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54546 (local address 127.6.46.252:33455)
0810 14:37:15.214437 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.214441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.215075 (+ 634us) server_negotiation.cc:407] Connection header received
0810 14:37:15.215214 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.215220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.215295 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.215377 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:15.216112 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54546: BlockingRecv error: recv got EOF from 127.0.0.1:54546 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 14:37:15.241933 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.236115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.6.46.253:41467)
0810 14:37:15.236320 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.236327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.236826 (+ 499us) server_negotiation.cc:407] Connection header received
0810 14:37:15.237005 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.237010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.237117 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.237230 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.237842 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.238793 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.240064 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.240493 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.240624 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.241170 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.241308 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.241385 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.241537 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.241641 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 14:37:15.249434 12643 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:15.247720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54548 (local address 127.6.46.252:33455)
0810 14:37:15.247898 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.247904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.248047 (+ 143us) server_negotiation.cc:407] Connection header received
0810 14:37:15.248268 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.248272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.248341 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.248419 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:15.249235 (+ 816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54548: BlockingRecv error: recv got EOF from 127.0.0.1:54548 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 14:37:15.294421 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.288883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.6.46.253:41467)
0810 14:37:15.289050 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.289055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.289734 (+ 679us) server_negotiation.cc:407] Connection header received
0810 14:37:15.289922 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.289926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.290044 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.290124 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.290525 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.291259 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.292649 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.293230 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.293336 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.293711 (+ 375us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.293813 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.293885 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.294053 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.294133 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 14:37:15.301941 12643 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:15.300390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54554 (local address 127.6.46.252:33455)
0810 14:37:15.300554 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.300559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.300580 (+ 21us) server_negotiation.cc:407] Connection header received
0810 14:37:15.300815 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.300819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.300903 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.300957 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:15.301782 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54554: BlockingRecv error: recv got EOF from 127.0.0.1:54554 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 14:37:15.375388 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.369974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40172 (local address 127.6.46.253:41467)
0810 14:37:15.370219 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.370226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.370727 (+ 501us) server_negotiation.cc:407] Connection header received
0810 14:37:15.370928 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.370935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.371004 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.371087 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.371510 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.372284 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.373618 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.374165 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.374284 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.374651 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.374789 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.374870 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.375016 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.375103 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
W20260810 14:37:15.382786 12643 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:15.380786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.6.46.252:33455)
0810 14:37:15.380989 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.380995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.381458 (+ 463us) server_negotiation.cc:407] Connection header received
0810 14:37:15.381615 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.381621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.381697 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.381776 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:15.382559 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54556: BlockingRecv error: recv got EOF from 127.0.0.1:54556 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 14:37:15.518596 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.514120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40178 (local address 127.6.46.253:41467)
0810 14:37:15.514335 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.514341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.514638 (+ 297us) server_negotiation.cc:407] Connection header received
0810 14:37:15.514831 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.514834 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.514904 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.514972 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.515310 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.515962 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.517162 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.517576 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.517663 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.517990 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.518115 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.518175 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.518266 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.518327 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260810 14:37:15.525154 12643 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 14:37:15.523698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.6.46.252:33455)
0810 14:37:15.523847 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.523851 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.524121 (+ 270us) server_negotiation.cc:407] Connection header received
0810 14:37:15.524254 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.524257 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.524326 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.524404 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 14:37:15.524996 (+ 592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54564: BlockingRecv error: recv got EOF from 127.0.0.1:54564 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 14:37:15.567879 12669 catalog_manager.cc:1259] Loaded cluster ID: a95d15fc07d2409fa6d7ec7c2104a3f5
I20260810 14:37:15.567975 12669 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e: loading cluster ID for follower catalog manager: success
I20260810 14:37:15.570500 12669 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e: acquiring CA information for follower catalog manager: success
I20260810 14:37:15.572194 12669 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 40f64b7412904560a5770681e7fb486e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 14:37:15.792234 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.787645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.6.46.253:41467)
0810 14:37:15.787842 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.787847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.788111 (+ 264us) server_negotiation.cc:407] Connection header received
0810 14:37:15.788282 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.788286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.788358 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.788448 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.788788 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.789469 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.790795 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.791196 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.791292 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.791661 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.791757 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.791819 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.791926 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.791991 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 14:37:15.802716 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.797764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.6.46.252:33455)
0810 14:37:15.797991 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.797995 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.798129 (+ 134us) server_negotiation.cc:407] Connection header received
0810 14:37:15.798291 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.798296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.798445 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.798533 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.798845 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.799530 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.800761 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.801142 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.801240 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.801658 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.802240 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.802348 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.802391 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.802452 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 14:37:15.814810 12526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.808158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.6.46.253:41467)
0810 14:37:15.808322 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.808327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.810131 (+ 1804us) server_negotiation.cc:407] Connection header received
0810 14:37:15.810303 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.810306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.810367 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.810438 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.810919 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.811715 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.813214 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.813589 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.813702 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.814115 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.814211 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.814286 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.814420 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.814494 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 14:37:15.816349 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.808595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35950 (local address 127.6.46.254:34457)
0810 14:37:15.809058 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.809063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.809077 (+ 14us) server_negotiation.cc:407] Connection header received
0810 14:37:15.809176 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.809180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.809317 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.809462 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.810150 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.811345 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.813767 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.814500 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.814598 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.815365 (+ 767us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.815784 (+ 419us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.815885 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.815994 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.816075 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":222,"threads_started":1}
I20260810 14:37:15.820225 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 14:37:15.808194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.6.46.252:33455)
0810 14:37:15.808335 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 14:37:15.808340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 14:37:15.814738 (+ 6398us) server_negotiation.cc:407] Connection header received
0810 14:37:15.814989 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 14:37:15.814995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 14:37:15.815068 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 14:37:15.815120 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 14:37:15.815657 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.816734 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.818475 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 14:37:15.819121 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 14:37:15.819243 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 14:37:15.819568 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 14:37:15.819696 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 14:37:15.819778 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 14:37:15.819861 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 14:37:15.819944 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 14:37:15.825603 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 12283
I20260810 14:37:15.826511 12387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:37:15.933765 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 12283
W20260810 14:37:15.959019 12558 connection.cc:535] server connection from 127.0.0.1:54476 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 14:37:15.959106 12437 connection.cc:535] server connection from 127.0.0.1:40062 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 14:37:15.959517 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 12413
I20260810 14:37:15.960582 12517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:37:16.064103 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 12413
I20260810 14:37:16.093513 6331 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task50NkZQ/build/asan/bin/kudu pid 12532
I20260810 14:37:16.094601 12633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 14:37:16.198516 6331 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task50NkZQ/build/asan/bin/kudu with pid 12532
2026-08-10T14:37:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4637 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9041 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (360975 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST